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This  thesis  reviews  the  decision-making  dilemma  caused 
by  information  uncertainty  and  ambiguity  produced  during 
crisis  situations.  Several  cross-impact  analysis  techniques 
are  reviewed  and  evaluated  as  possible  crisis  decision-aids. 
Cross-event  analysis  is  selected  and  extended  for  demonstra- 
tion in  a hypothetical  crisis  situation  involving  South 
Asia.  The  selected  technique  is  operationalized  and  employed 
in  a controlled  environment  to  assess  policy  response  options 
to  the  hypothetical  crisis.  The  thesis  then  assesses  the 
techniques' 3 conceptual  limitations,  and  evaluates  its  utility 
as  a potential  crisis  decision-aiding  methodology. 
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I.  DECISION-MAKING  IN  CRISIS  SITUATIONS 


A.  INTRODUCTION 

Accounts  describing  foreign  policy-making  indicate  decision- 
makers frequently  deal  with  profuse  quantities  of  intelligence 
information}  which  may  be  inconclusive,  ambiguous,  or 
conflicting.  In  an  analysis  of  the  Pearl  Harbor  surprise 
attack,  Wohlstetter  reveals  problems  resulting  from  infor- 
mation quantity  and  unclear  meaning.  She  characterizes  the 
situation  as  one  in  which,  "...the  mass  of  signals  grow 
increasingly  dense  and  freighted  with  ambiguities."  [Ref.  1, 
p.  3).  Sorensen  points  out  that  a recurring  problem  facing 
Presidents  is  that  the  facts  concerning  a situation  "may 
be  in  doubt  or  dispute."  [Ref.  2,  p.  19).  Schlesinger  high- 
lights the  ambiguity  surrounding  the  1963  Vietnam  situation 
by  observing  that  the  President  asked;  "Were  you  two  gentle- 
men in  the  same  country?"  [Ref.  3,  p.  993],  upon  receiving 
two  widely  divergent  situation  assessments  from  two  officials 
sent  to  Vietnam  on  a fact-finding  mission.  This  story  also 
demonstrates  the  important  role  individual  perceptions  play 
in  decision-making. 

Analysts  suggest  perceptions  influence  the  decision 
process  because  decision-makers  interpret  information  using 
their  insights,  expectations,  experience,  and  goals.  Sprout 
and  Sprout  refer  to  this  as  the  "stateman’s  psychological 
environment  (that  is  his  image,  or  estimate,  of  the  situation. 
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setting,  or  milieu)  [which]  may  or  may  not  correspond  to 
the  operational  environment"  [Ref.  4,  p.  318].  Perceptions 
are  a key  factor  in  recognizing  crisis  situations  [Ref.  5]. 

Yet,  it  is  in  just  such  situations  that  information  overloads 
and  inconsistencies,  which  may  confound  a decision-maker's 
perceptions,  seem  to  occur  most  dramatically  [Ref.  6]. 

This  chapter  develops  the  concept  that  a "dilemma"  exists 
in  decision-making  resulting  from  information  overload  and 
ambiguity.  Establishing  the  dilemma's  relationship  to 
crises,  and  establishing  its  effect  on  the  decision  process 
requires  introducing  several  relevant  concepts.  These  include, 
identifying  a basic  decision  process  model,  defining  the 
concept  of  crisis,  identifying  its  stress  and  information 
overload  effects  on  decision-making,  and  examining  factors 
which  may  reduce  the  dilemma.  The  purpose  is  to  establish 
the  need  for  a decision-aiding  methodology  which  can  lessen 
the  dilemma's  impact  on  the  policy-making  process. 

B.  THE  DECISION-MAKING  PROCESS 

According  to  general  decision-making  theory,  as  developed 
by  Turban  and  Meredith  [Ref.  7],  the  first  step  in  the 
decision-making  process  is  problem  identification.^  The  same 


Subsequent  steps  include:  searching  for  the  alternative 
courses  of  action,  evaluating  the  alternatives,  and  solution 
selection.  These  steps  were  not  examined  in  detail  but 
should  be  kept  in  mind  during  the  discussion  of  the  approaches 
to  foreign  policy  analysis. 


source  suggests  identifying  problems  through  comparing 
desired  and  actual  levels  of  goal  attainment.  This  pre- 
supposes that  organizational  goals  are  explicit,  and  in  a 
form  which  facilitates  measuring  their  level  of  accomplishment. 

Such  a requirement  may  be  met  in  product  oriented  organi- 
zations (capturing  a specific  share  of  the  market,  or 
obtaining  a specific  profit  level)  however,  it  becomes  much 

more  difficult  in  organizations  which  frame  goals  in  terms 

2 

of  advancing  or  protecting  the  nation's  interests.  Thus, 
in  starting  the  decision  process,  the  policy-maker  immediately 
faces  a problem  using  available  information  to  determine  what 
national  interests  an  emerging  situation  threatens.  The 
available  information's  accuracy  greatly  influences  the  pre- 
cision of  nation:ki  interest  identification  and  measurement. 

Turbin  and  Meredith  [Ref.  7]  propose  using  an  evalua- 
tion shceme  in  completing  the  decision  process.  The  evalua- 
tion scheme  requires  some  technique  which  predicts  alternative 
response  outcomes,  a means  of  relating  outcomes  to  goals, 
and  a decision  rule  applicable  to  response  selection.  The 
authors  also  describe  factors  which  complicate  the  decision 
process . 


2 

This  problem  directly  relates  to  the  requirement  for 
a method  of  system  monitoring  which  would  be  basic  to  a 
decision-aiding  system.  This  aspect  of  the  aid  was  not 
developed  extensively  in  the  current  study  due  to  its  com- 
plexity and  the  time  required  to  develop,  and  validate 
indicators.  Significant  research  is  being  conducted  in  this 
area  by  C.A.  McClelland  and  others  in  the  Threat  Recognition 
and  Analysis  Project  [Ref.  8]. 
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Complications  develop  in  decision-making  when  incorpora- 


ting multiple  goals,  evaluating  an  indefinite  number  of 
alternatives,  and  determining  how  sensitive  outcomes  are 
to  changes  in  problem  configuration.  (Changes  in  problem 
configuration  could  result  from  incomplete  or  inaccurate 
information.  These  slight  changes  may  alter  a policy  alterna- 
tive's impact  on  overall  situational  outcomes,  and  require 
an  analysis  of  the  sensitivity  of  the  policy  option's  effect 
to  changing  circumstances.)  The  above  problems  obviously 
exist  in  the  policy  decision  process.  This  study's  major 
premise  is  that  an  information  processing  dilemma,  beyond 
the  range  of  these  problems,  further  complicates  policy 
decision-making  in  a crisis  situation.  The  characteristics 
defining  a crisis  situation  produce  this  dilemma. 

C.  THE  CONCEPT  OF  CRISIS 

Traditional  international  relations  studies  often  ex- 
plore the  relationship  between  "crisis”  and  "normal"  inter- 
actions. These  works,  usually  describing  events  surrounding 
a crisis,  give  the  concept  several  intuitive  interpretations. 

The  most  common  among  these  view  crises  as  marking  "turning 
points"  in  international  relations.  Other  interpretations 
describe  crises  as  events  which  occur  naturally  within  the  system.^ 


In  the  Marxist-Leninist  perspective,  the  occurrence  of 
international  crisis  is  an  expression  of  the  process  of  world 
revolution.  Traditional  power  theory  explains  crisis  as  the 
operation  of  the  balance  of  power.  Here,  crisis  results  from 
changes  in  the  distribution  of  power,  or  from  policy  makers 
applying  "field-tests"  to  determine  the  status  of  the  balance 
[Ref.  9]. 


In  his  essay  on  "The  Acute  International  Crisis,"  McClelland 

reviews  these  interpretations  and  concludes: 

The  crises  are  thought  to  be  connected 

with  the  international  struggles  for 

ideas  and  also  for  power.  They  are  seen 

as  indicators  of  the  state  of  the 

international  system,  but  are  regarded 

as  prime  operations  in  the  system  as 

well.  They  seem  to  be  related  to  going 

to  war  and  to  staying  at  peace.  (Ref.  9,  p.  188). 

Obviously,  crisis  is  a pivotal  concept  in  international  rela- 
tions. The  problem  is  to  establish  it  as  a distinct  phenomenon 
with  recognizable  character istics . 

Recent  theoretical  approaches  to  international  rela- 
tions studies  attempt  to  operationalize  the  concept  of 
crisis.  These  approaches  are  examined  in  a later  section 
developing  the  dilemma's  impact  on  the  policy  decision  pro- 
cess. The  approach  advanced  by  Hermann  (Refs.  10  and  11] 
was  selected  as  most  relevant  to  the  current  study.  He  des- 
cribes three  dimensions  which  bound  a crisis  situation. 

1 . High  Threat 

The  first  dimension  defining  crisis  arises  because 
the  situation  poses  "a  potential  hindrance  or  obstruction  to 
some  object  or  state  of  affairs  that  a decision-making  unit 
is  motivated  to  achieve."  (Ref.  10,  p.  29).  A situation 
that  threatens  important  goals  produces  fear  that  failure 
to  make  a "good"  decision  could  leave  these  goals  in  jeopardy. 
Having  accurate  and  timely  information,  therefore,  becomes 
extremely  important.  The  other  dimensions  bounding  the 
situation  make  it  less  likely  such  information  is  available. 
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Short  Response  Time 


1 

I ■ 


f Combining  with  the  high  threat  factor  is  the  real- 

I ization  that  in  a short  period  of  time,  if  no  action  is 

t 

i taken,  the  situation  may  change  in  some  major  (unfavorable) 

1 • 

way.  This  aspect  requires  reaching  a decision  quickly,  al- 
I lowing  little  time  for  collecting  information.  Furthermore, 

I 

I short  response  time  does  not  permit  establishing  elaborate 

\ I 

■ search  routines  to  disclose  alternative  response  options. 

Thus,  information  currently  available  forms  the  basis  for 
the  decision.  While  decision-makers  may  receive  more  informa- 
tion, its  accuracy  is  questionable,  and  individual  percep- 
tions become  more  important  in  determining  what  policy  actions 
are  eventually  taken.  The  third  dimension  limits  the  stored 
information's  utility. 

3 . Surprise 

This  final  dimension  results  from  the  "absence  of 
awareness  on  the  part  of  policy  makers  that  the  situation 
is  likely  to  occur."  [Ref.  10,  p.  30).  This  surprise  aspect 
may  stem  from  incomplete  information,  or  result  from  a 
policy-maker's  satisfaction  with  the  situation,  producing 
a reluctance  to  believe  contrary  reports.  Surprise  makes 
it  unlikely  advance  preparations  exist  for  coping  with  the 
situation,  producing  an  "ad-hoc"  decision  process.  Surprise 
also  prevents  serious  data-collection  which  usually  precedes 
significant  anticipated  events  [Ref.  11,  p.  4161.  These 
characteristics  produce  a dilemma  which  inhibits  effective 
crisis  response. 
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D.  THE  DILEMMA 


In  responding  to  a developing  crisis,  policy-makers 
face  a dilemma  dealing  with  breakdowns  in  what  information 
usually  means.  As  a crisis  situation  emerge,  policy-makers 
receive  unanticipated  information  inputs.  These  inputs  are 
unexpected  because  they  differ  from  perceptions  concerning 
"normal"  information  outputs  which  the  situation  should  be 
generating.  Thus,  the  dilemma's  first  aspect  involves  un- 
derstanding seemingly  "unique"  events.  Policy-makers  must 
correctly  assess  this  information  to  realize  that  a problem 
exists,  and  to  determine  its  importance. 

As  the  crisis  continues,  more  unique  inputs  flow  in, 
increasing  the  information  load  the  policy-maker  must  pro- 
cess [Ref.  12].  This  aspect  ta,xes  the  decision  units' 

4 

ability  to  sort  and  rapidly  interpret  available  information. 
Janis  and  Mann  (Ref.  14]  identify  several  compensating  proce- 
dures individuals  use  in  coping  with  information  overload. 


4 

A large  body  of  literature  exists  indicating  that  in- 
dividuals are  sub-optimal  information  processors.  Fisher, 
et  al.  [Ref.  13]  review  the  literature  on  individual's  in- 
formation processing  abilities.  Their  report  indicates  that 
unaided  human  judgements  in  complex  decision  tasks  are  often 
less  accurate  than  formal  algorithms.  For  instance,  evidence 
presented  indicates  that  individuals  are  conservative  in 
interpreting  data  due  to  misperception  and  misaggregation. 

This  implies  that  decisions  are  normally  based  on  a small 
number  of  information  items,  thus  excess  data  and  effort  spent 
in  its  collection  is  "wasted".  Other  evidence,  collected 
from  "real-world"  studies,  indicates  that  decision-makers 
have  difficulty  using  multiple  cue  information. 

The  current  study  e-x^unines  the  way  in  which  crisis 
situations  further  reduce  decision-makers'  capabilities  to 
utilize  information.  The  Fisher  report  indicates  that  the 
existence  of  the  basic  problem  is  enough  to  justify  a 
decision-aiding  system. 
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Many  of  these  inhibit  the  accurate  evaluation  of  response 
alternatives. 

Evaluation  also  requires  dependable  information.  Un- 
fortunately, information  during  a crisis  may  be  inaccurate 
or  conflicting  due  to  source  unavailability  or  distortion. 

Many  factors  might  cause  this  distortion;  however,  the 
effect  is  to  reduce  useful  information.  The  policy  maker 
must  discriminate  between  relevant  and  irrelevant;  reliable 
and  unreliable  information.  A crisis  situation's  high 
threat  and  time  pressure  components  imapir  the  discrimina- 
tion ability  which  information  ambuiguity  requires. 

One  of  this  study's  central  propositions  is  that;  Policy- 
makers in  a crisis  situation  must  struggle  to  interpret 
large  quantities  of  unusual  data,  having  ambiguous  or  con- 
flicting meaning.  Because  the  situation  involves  high 
threat,  the  perception  develops  that  the  decision  is  crucial. 
The  rapid  event  flows  require  accurate  information  evaluation, 
and  allow  a limited  decision  time.  The  situation's  surprise 
characteristic  precludes  utilizing  routine  response  measures. 

To  interpret  large  quantities  of  ambiguous  data  the  policy- 
maker must  rapidly  and  efficiently  focus  all  available  informa- 
tion and  expertise  on  the  situation. 

Reviewing  current  approaches  to  foreign  policy  analysis 
indicates  that  this  dilemma  affects  the  entire  policy-making 
process.  This  review  reveals  the  requirement  for  a real 
time  decision-aiding  methodology  which  can  reduce  the  infor- 
mation dilemma's  impact  on  policy  formulation. 
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E.  FOREIGN  POLICY  ANALYSIS 

Traditional  foreign  policy  studies  developed  the  found- 
ations for  more  systematic  approaches  in  identifying  threats 
to  national  interests  (crises) . Hermann  [Ref.  10]  and 
McClelland  (Ref.  9]  review  the  intuitive  interpretations  of 
crisis  found  in  descriptive  international  relations  studies. 
The  early  works  these  authors  review  describe  crises  as 
"turning  points"  in  international  relations. 

Crises  were  believed  to  result  from  competing  national 
interests,  or  as  situations  which  developed  from  a series 
of  events  a nation  viewed  as  somehow  posing  a threat  to  its 
interests.  The  effort  to  develop  a more  exact  understanding 
of  foreign  policy  and  crises  produced  two  broad  analytic 
approaches.  These  approaches  can  be  classified  "systemic" 
and  "decisions;"  both  advance  theoretical  arguments  and 
research  findings  revealing  the  policy  maker's  dilemma. 

Parker  [Ref.  15]  describes  the  tenets  of  these  two  approaches 
and  their  major  contributions. 

A third  approach  stems  from  applying  information  theory 
to  foreign  policy  analysis,  contributing  new  insights  about 
communications  flows  in  policy-making.  This  technique 
attempts  to  quantify  concepts  such  as  uncertainty  and  "noise." 

1.  Systems  Approach 

The  systems  approach  views  crisis  as  an  indication 
of  the  overall  state  of  the  system,  directly  related  to 


I 

i 


17 


The  connection  to  the  traditional 


system  stability.^ 

intuitive  interpretations  is  obvious.  From  this  viewpoint  it 
can  be  argued  the  occurrence  of  crises  can  be  used  to  iden- 
tify the  emergence  of  a problem  situation. 

An  alternative  view  of  crisis  in  systems  studies 
proposes  that  it  is  a prime  operation  in  the  system  itself. 

This  view  utilizes  March  and  Simon's  bureaucratic  process 
model  [Ref.  171.  In  this  model,  organizations  (modernizing 
societies)  strive  for  stability  in  their  environment.  Recog- 
nizing a problem  as  having  been  dealt  with  previously,  the 
organization  tends  to  employ  repetitive  responses.  As  a 
result  the  organization  establishes  routine  procedures  when- 
ever possible.  Crises,  in  this  model,  are  complexes  of 
events  overflowing  normal  processing  channels,  forcing  new 
outputs,  which  return  to  the  environment  producing  new 
inputs  until  new  solutions  are  found  [Ref.  9].  This  sequence 
demonstrates  how  information  ambiguity  and  uncertainty  impact 
the  decision  process.  1 

2.  Decisions  Approach  ] 

The  second  major  approach  to  foreign  policy  analysis 
suggests  that  a stimulus-response  relationship  exists  be-  ^ 

tween  crisis  and  decision  [Ref.  11].  This  approach  highlights 

' 

^McClelland  [Ref.  9]  derives  this  interpretation  from 
Hans  J.  Morgenthau's  Politics  Amounq  Nations.  Price  [Ref. 

161  proposes  that  system  stability  and  decision  unit  flexi-  , 

bility  combine  to  define  the  range  of  options  decision- 
makers consider  in  a crisis  situation. 


the  importance  that  decision-makers'  perceptions  play  in 
responding  to  incoming  data.^ 

The  decisions  approach  identifies  crisis  as  a sit- 
uational variable  affecting  the  decision  process  in  two 
ways.  These  result  from  crisis'  stress  inducing  character- 
istics, and  their  impact  on  information  processing.  These 
factors  are  closely  interrelated  but,  for  reasons  of  clarity, 
are  explored  separately  in  this  study. 

Hermann  [Ref.  10]  proposes  that  crises  are  situations 
involving  high  threat,  allowing  short  response  time,  and 
surprising  the  decision  unit.  Holsti  [Ref.  6],  and  other 
authors  [Ref.  14] , present  psychological  evidence  demonstrating 
that  these  characteristics  are  stress  inducing.^  The  findings 
indicate  that  crisis  induced  stress  reduces  decision-making 
capabilities,  especially  those  involved  in  identifying, 
sorting,  processing,  and  interpreting  information. 

Foreign  policy  studies  [Ref.  6],  and  field  experi- 
ments [Ref.  10] , give  credence  to  these  conclusions.  These 


Sprout  and  Sprout  fully  develop  this  concept  in  their 
works  [Ref.  4 and  18].  Holsti,  et  al,  [Ref.  5]  modeled  the 
process  utilizing  this  factor.  The  model  included:  (1)  the 
stimulus  (meauiing  policy)  applied  by  the  initiating  state, 

(2)  the  perception  of  the  stimulus  by  the  receiving  state's 
decision  unit,  (3)  the  receiving  state's  policy  response,  and 
(4)  response  perception  by  the  initiating  state's  decision  unit. 

high  degree  of  threat  to  important  values  produces 
anxiety  and  overreaction.  Long  working  hours  required  by 
short  response  times,  and  the  perception  of  that  pressure, 
magnify  stress.  Unanticipated,  novel  situations  (surprise) 
are  generally  more  threatening  to  decision-makers  [Ref.  6). 
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findings  indicate  that  policy  makers  face  a dilemma  inter- 
preting and  using  information  throughout  the  entire  decision 
process.  Finally,  indications  are  that  stress  is  self- 
reinforcing,  suggesting  that  the  dilemma  increases  as  the 
situation  continues  [Ref.  6]. 

These  studies  reveal  how  stress  affects  the  decision 
process  by  reducing  the  decision  unit's  ability  to  process 
information.  One  study  went  further  and  excimined  informa- 
tion volume  produced  during  the  crisis  leading  to  World 

War  I [Ref.  6].  Holsti's  findings  indicate  that  both  infor- 

0 

mation  flow  and  uncertainty  rose  during  the  pre-war  period. 
This  increases  both  stress  and  information  overlaod  [Ref.  14] . 
3 . Information  Theory 

Using  information  theory  in  foreign  policy  analysis 
is  a relatively  new  approach  which  clarifies  the  relation- 
ship between  uncertainty  and  stress.  It  measures  information 
levels  and  uncertainty  contained  in  transmitted  messages. 
Studies  using  this  technique  report  findings  which  relate  to 
interpreting  incoming  intelligence  information. 

Lee  [Ref.  21]  explains  information  measurement,  and 
demonstrates  that  "noise"  in  the  system  produced  by  erroneous 


p 

Other  studies  have  explored  the  role  of  uncertainty 
in  the  occurrence  of  war.  Singer,  et  al.  [Ref.  19]  conclude 
that  the  changing  role  of  uncertainty  accounts  for  the  differ- 
ent time  spans  for  which  their  models  predicted  the  occurrence 
of  conflicts.  An  attempt  to  formalize  the  procedure  used 
in  the  Singer  study  demonstrated  the  potential  importance 
of  uncertainty  in  the  decision  process  [Ref.  20]. 
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or  conflicting  information  reports  significantly  reduces 
a message's  information  value.  McClelland  [Ref.  22]  uses 
information  theory  in  developing  a measure  of  uncertainty, 
which  he  employed  in  studying  the  Berlin  question.  This 
value  represents  uncertainty  levels  present  in  a given  sit- 
uation. This  uncertainty  reflects  analysts'  efforts  to 
respond  to  the  other  side's  actions.  Uncertainty,  measured 
in  terms  of  H-Rel  values,  increased  fairly  sharply  during 

9 

"obvious"  crisis  phases  in  the  Berlin  problem. 

These  findings  demonstrate  that  policy  decisions  are 
made  under  uncertainty  and  increased  information  volume. 

This  situation  strains  the  decision-maker's  ability  to  cope 
with  an  unfolding  situation.  Threat,  time  pressure,  and 
surprise  combine  to  increase  stress,  and  further  degrade 
decision-making  capabilities. 

The  preceding  discussion  explored  the  information 
dilemma  facing  policy  makers.  This  dilemma  involved  three 
aspects:  (1)  responding  to  unexpected  information,  (2)  coping 

with  increased  information  volume,  and  (3)  interpreting 
ambiguous,  conflicting  reports.  Because  the  dilemma  reduces 
decision-making  capabilities,  the  requirement  exists  for 
a decision-aid  to  reduce  its  impact  on  the  decision  process. 


9 

H-Rel  is  defined  as  the  ratio  of  absolute  uncertainty 
(some  knowledge  of  performance)  to  maximum  uncertainty  (no 
performance  data  known).  As  this  ratio  approaches  1.00, 
it  suggests  that  the  behaviors  have  shown  increasing  signs 
of  disorderliness,  or  indicates  a large  amount  of  "variety" 
in  event  emissions  (Ref.  22,  p.  172] . 
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To  be  useful  such  an  aid  must  perform  several 
functions . 

F.  REDUCING  THE  DILEMMA 

The  problems  of  multiple  goals,  evaluating  an 
indefinite  number  of  options,  possible  slight  changes 
in  problem  configuration,  and  the  need  to  incorporate  new 
inforination  complicates  the  policy  decision  process  (Ref. 

7],  A useful  decision-aid  must,  therefore,  react  to  changing 
circumstances  and  incorporate  diverse  inputs. 

Information  relevant  to  a particular  situation  is 
often  incomplete,  outdated,  or  inaccurate.  The  decision- 
maker must,  therefore,  be  able  to  draw  on  data  other  than 
information  about  the  immediate  situation  [Ref.  11] . The 
perceptions  and  judgments  of  specialists  in  the  field  might 
be  one  source  of  such  data.  The  decision-aid  should  focus 
specialists'  expertise  and  knowledge  to  extract  meaning  from 
incomplete  information. 

A true  decision-aiding  system  - as  opposed  to  an  infor- 
mation processing  system  - should  help  reduce  the  problem. 

It  should  facilitate  problem  identification,  aid  in  enumera- 
ting options,  and  help  determine  their  potential  outcomes. 

It  should  also  provide  the  means  to  evaluate  outcomes,  and 
indicate  optimum  responses. 

The  short  response  time  inherent  in  a crisis  situation 
requires  rapid  yet  considered  action.  To  reduce  decision 
time  and  allow  thorough  evaluation,  the  decision-aid  should 

1 
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be  computer  adaptable.  This  capability  would  permit 
obtaining  interactions  from  numerous,  geographically  separate 
analysts  and  specialists.  It  would  also  provide  the  capa- 
bility for  conducting  sensitivity  analysis,  and  hard  copy 
results  for  later  analysis. 

G.  OBJECTIVES 

This  research  project  will  develop  and  evaluate  a poten- 
tial real  time  decision-aiding  technique  to  examine  policy 
options  for  response  in  a crisis  situation.  A decision 
analysis  methodology  which  deals  with  alternative  evaluation 
under  uncertainty  will  be  selected  as  the  basis  for  the 
potential  decision-aid. 

After  determining  the  methodology's  information  input 
requirements,  working  assumptions,  limitations,  and  alterna- 
tive operation  procedures,  the  methodology  will  be  adapted 
into  a policy  analysis  format.  Designing  a policy  analysis 
format  requires  structuring  the  methodology  to  identify  and 
rank  national  objectives  affected  by  a specific  problem,  to 
identify  possible  response  options  supporting  those  objectives, 
to  determine  each  option's  contribution  to  achieving  those 
objectives,  and  to  evaluate  their  relative  merit. 

The  modified  technique  will  then  be  demonstrated  in  a 
controlled  environment.  A study  group,  representing  experts, 
will  be  presented  with  a hypothetical  crisis  situation 
in  which  the  objective  is  to  determine  an  effective  response. 
The  technique  will  be  used  to  guide  the  group  in  their 
consideration  of  the  problem,  and  to  correlate  their  inputs. 
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^ Based  on  this  demonstration,  the  analytic  technique 
will  be  evaluated  on  its  effectiveness  in  structuring  and 
obtaining  a solution  to  the  problem.  The  technique's  util- 
ity will  be  assessed  regarding  its  flexibility,  ease  of 
* 

application,  performance,  and  potential  for  computerization. 
H.  CONCLUSIONS 

A crisis  is  defined  by  three  "situational"  boundaries 
[Refs.  10  and  11] . 

1.  High  threat  to  decision  unit  goals. 

2.  Short  response  time  available. 

3.  Surprising  the  decision  unit  by  its  occurrance. 

These  dimensions  impact  the  decision  process  producing  three 
complicating  factors  which  reduce  a decision-maker's  already 
questionable  abilities  [Ref.  13]  to  perform  effectively  when 
it  is  vitally  important  to  do  so. 

These  factors  are  [Refs.  6 and  22] : 

1.  Stress 

2.  Uncertainty 

3.  Information  Overload 

Increases  in  the  levels  of  these  factors,  combined  with  poor 
information  processing  capabilities,  produces  a dilemma  in 
interpreting  information  concerning  a rapidly  evolving  situa- 
tion . 

This  dilemma  reduces  the  decision-maker's  ability  to: 

1.  Identify  relevant  useful  data. 

2.  Correlate  and  interpret  infromation. 

3.  Use  this  information  to  draw  accurate  inferences, 
and  make  responsible  judgments. 

4.  Determine  appropriate  response  alternatives,  and 
evaluate  their  utility. 

5.  Make  a logical  solution  selection. 
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Reducing  this  dilemma  to  improve  decision-making 
abilities  requires  a decision-aid  which  is; 

1.  Flexible  enough  to  incorporate  multiple  goals, 
diverse  options,  and  incomplete  information. 

2.  Draws  on  the  "stored  information"  of  an  expert's 
perceptions  to  interpret  the  situation. 

3.  Quickly  focuses  the  expertise  and  knowledge  of 
numerous  specialists  on  a situation  to  evaluate 
possible  responses. 

4.  Systematically  evaluates  alternatives  and  indicates 
an  optimal  response. 

5.  Is  computer  adaptable. 

Subsequent  chapters  of  this  study  will  identify  and 
examine  an  existing  decision  analysis  technique,  develop 
it  into  a possible  policy  analysis  tool,  demonstrate  its 
application,  and  evaluate  its  utility. 
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II.  CROSS- IMPACT  ANALYSIS 


A.  INTRODUCTION 

The  previous  chapter  identified  problems  facing  decision- 
makers operating  in  an  environment  characterized  by  uncertainty 
and  ambiguous  information.  In  a social  or  political  environ- 
ment, moving  from  problem  awareness  to  effective  control  or 
corrective  actions  requires  that  decision-makers  understand 
the  complex  relationships  operating  in  the  environment.  Such 
understanding  is  inhibited  when  a problem's  components  are 
defined  and  related  through  imprecise  concepts  and  incomplete 
theories . 

As  indicated  earlier,  social  and  political  problems  are 
not  easily  adaptable  to  systematic  "scientific"  analysis. 

In  many  cases,  the  problem  elements  are  not  adequately  defined 
nor  operationalized.  Furthermore,  no  adequate  theory  exists 
which  describes  the  complex  relationships  among  problem  ele- 
ments and  indicators.  These  difficulties  are  apparent  in 
the  attempts  to  construct  theories,  define  variables,  and 
identify  reliable  indicators  for  crises  reviewed  in  Chapter 
One. 

Socio-economic  and  political  problems  are  defined  using 
imprecise  concepts  based  on  a limited  understanding  of  the 
relationships  aunong  the  variables  involved.  For  this  reason, 
conducting  systematic  decision  analysis  requires  an  alterna- 
tive approach  to  problem  specification.  The  basis  for  one 
alternative  approach  is  suggested  by  Helmer: 
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. . .we  can  either  throw  up  our  hands 
in  despair  and  wait  until  we  have  an 
adequate  theory  enabling  us  to  deal 
with  socio-economic  problems  as  con- 
fidently as  we  do  with  problems  in 
physics  and  chemistry,  or  we  can 
make  the  most  of  an  admittedly  unsat- 
isfactory situation  and  try  to  obtain 
the  relative  intuitive  insights  of 
experts  and  then  use  their  judgments 
as  systematically  as  possible. 

[cited  in  Ref.  23,  p.  2) 

The  Delphi  Technique  was  an  early  approach  to  employing 
subjective  judgments  in  problem  evaluation.  It  was  sub- 
jected to  several  criticisms,  one  being  that  it  produced 
"linearly  independent"  estimates  which  failed  to  account 
for  potential  interactions  among  the  events  being  considered 
[Ref.  24].  Cross-Impact  analysis  is  a Delphi  extension 
intended  to  overcome  this  problem. 

This  chapter  will  present  a non-technical  explanation  of 
the  fundamental  notions  underlying  cross-impact  analysis. 

It  will  identify  and  classify  contentious  issues  which  must 
be  resolved  before  the  technique  can  be  used.  This  discussion 
is  intended  to  provide  a basis  for  systematically  comparing 
the  various  approaches  to  conducting  cross-impact  analysis. 
Specific  areas  which  will  be  explored  include:  (1)  the  tech- 
nique's basic  concepts,  (2)  its  information  requirements 
and  their  interpretations,  (3)  the  computational  procedure 
used,  and  (4)  the  technique's  outputs  and  possible  applica- 
tions . 

In  their  review  of  the  literature  on  decision-making, 
Fischer,  et.  al.  examine  several  decision-aiding  technologies 


i 

*«- 


: f 


27 


which  use  subjective  estimates.  These  technologies  follow 
a "decomposition"  approach  involving  six  tasks: 

1.  Recognizing  that  a decision 
problem  exists. 

2.  Identifying  the  possible 
courses  of  action. 

3.  Constructing  a probabilistic 
model  of  the  decision-making 
environment. 

4.  Constructing  a model  to  eval- 
uate the  possible  consequences 
of  each  available  action. 

5.  Selecting  a course  of  action. 

6.  Implementing  the  alternative 
selected. 

[Ref.  13,  p.  43-44]  . 

Because  steps  1,  2,  and  6 involve  understanding  the 
creative  and  organizational  processes  inherent  in  decision- 
making, most  decision-aiding  technologies  concentrate  on 
steps  3,  4,  and  5.  As  this  study  will  show,  cross-impact 
analysis  fits  in  well  with  these  techniques . Cross-impact 
analysis  is  similar  to  other  subjective  techniques  in  that: 
it  develops  an  event  set  for  defining  the  problem  environment, 
it  constructs  a probabilistic  model  of  that  environment,  and 
it  uses  the  model  to  evaluate  possible  courses  of  action. 


It  should  be  noted  that,  on  occasion,  an  entire  commun- 
ity's evaluation  can  "miss  the  mark".  The  recent  events  in 
Iran  provide  an  example.  Reportedly  the  entire  intelligence 
community  was  surprised  at  the  situation's  rapid  deteriora- 
tion (two  crisis  dimensions)  [Ref.  25).  A large  body  of 
psychology  and  organizational  information  processing  litera- 
ture addresses  the  causes  of  this  phenomenon.  These  causes 
are  outside  this  study's  scope,  which  attempts  to  develop 
and  evaluate  an  aid  for  reducing  the  information  dilemma. 
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B . THE  CONCEPT 


Cross-impact  analysis  is  a technique  which  attempts  to 
identify  a given  situation's  possible  outcomes  by  quantifying 
the  linkages  between  potential  developments.  This  is  accom- 
plished by  systematically  assessing  how  any  particular  event 
impacts  all  other  relevant  events.  The  technique  and  its 
various  applications  are  based  on  the  realization  that  moat 
events  are  connected  in  some  way  (Ref.  26].  As  explained 
by  Gordon  and  Hayward:  "it  is  hard  to  imagine  an  event 
without  a predecessor  which  made  it  more  or  less  likely  or 
influenced  its  form  - or  one  which,  after  occurring,  left 
no  mark."  They  term  these  possible  interrelationships  "cross- 
impacts" (Ref.  26,  p.  101]. 

A second  principle  behind  this  technique  is,  that  due 
to  the  nature  of  the  "inexact  sciences",  the  only  possible 
approach  to  quantifying  these  interrelationships  is  to 
employ  an  e.xpert's  subjective  judgments  as  a "substitute 
for  the  exact  laws  of  causality  found  in  the  physical  sciences." 
(Ref.  24,  p.  134] . When  the  technique  is  employed,  estimates 
are  obtained  for  three  event  aspects  hypothesized  to  govern 
possible  interactions  among  future  developments:  (1)  Mode  - 
whether  an  event  "enhances"  or  "inhibits"  another;  (2)  Strength 
how  strong  an  impact  one  event  has  on  another;  and  (3)  Time 
Lag  - how  long  before  an  event's  influence  is  felt  by  other 
events.  These  aspects  are  developed  in  Gordon,  Enrer,  and 
Martino  (Refs.  26,  27,  and  28].  While  latter  cross-impact 
approaches  do  not  assess  these  aspects  explicitly,  understanding 
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the  interaction  concept  is  important  to  understanding  the 
intention  and  approach  behind  the  analysis. 

As  a corollary  to  using  subjective  estimates,  group 
opinions  are  usually  obtained  on  the  basis  that;  "The 
accuracy  of  a group  judgment  is  always  greater  than  (or  at 
least  equal  to)  the  average  accuracy  of  the  individual 
judgments."  (Dalkey,  cited  in  Ref.  29,  p.  140].^^  Fischer, 
et.  al.  report  several  other  studies  which  support  this 
conclusion  [Ref.  13,  p.  50]. 

C.  REQUIREMENTS  FOR  CONDUCTING  CROSS -IMPACT  ANALYSIS 

To  develop  a probabalistic  model  of  the  problem  environ- 
ment using  cross-impact  analysis  requires  that  four  proce- 
dures be  accomplished.  The  first  three  encompass  the  sub- 
jective estimation  process  and  include:  (1)  specifying  an 
event  set  which  defines  the  environment,  (2)  estimating  the 
initial  occurrence  probabilities  of  those  events,  and  (3) 
estimating  their  interactions  with  each  other  (or  their 
cross-impacts) . The  final  procedure  manipulates  the  model 
formed  by  these  estimates  to  produce  the  data  with  which 
possible  courses  of  action  are  evaluated.  This  step  requires 
(4)  developing  am  algorithm  to  use  in  the  manipulation  process. 
These  procedures  will  be  explained  in  detail  below. 


There  are  a number  of  subjective  measurement  techniques 
which  depend  upon  aggregating  group  estimates.  Subjective 
measurement,  aggreation  procedures  and  applications  are  dis- 
dussed  in  Torgerson,  Miller,  and  Coombs  [Refs.  30,  31,  and  32]. 
The  commonly  used,  group-based  subjective  measurement  techniques 
rely  heavily  on  central  tendencies  (i.e.,  means  and  standard 
deviations)  in  order  to  determine  judgmental  scores. 


1.  Event  Set  Specification 


The  first  process  undertaken  in  cross-impact  analy- 
sis is  developing  an  event  set.  This  is  "a  set  of  possible 
events  which  are  thought  to  be  important  to  the  issue  being 
explored"  [Ref.  33,  p.  33].  The  event  set  may  be  obtained 
in  any  manner:  delphi,  conferences,  literature  searches, 
consultations  with  experts,  analyst  specification,  and 
so  forth.  A single  event  may  address  any  subject,  and  the 
entire  set  may  address  any  combination  of  subjects  such  as, 
"potential  technological  developments,  a list  of  social  or 
political  developments,  a combination  of  these,  or  something 
entirely  different."  [Ref.  34,  p.  344). 

As  broad  and  loose  as  these  requirements  may  seem, 
the  means  by  which  the  event  set  was  formulated  in  several 
of  the  demonstrations  discussed  in  this  study  has  been  criti- 
cized as  faulty  [Ref.  29,  p.  137].  Therefore,  it  seems 
relev6mt  to  review  recommendations  for  accomplishing  this 
process. 

Enzer  suggests  candidate  events  be  developed  from  the 
likely  consequences  of  inaction,  or  allowing  the  "status-quo" 
to  continue  [Ref.  27,  p.  50].  He  also  proposes  several 
criteria  for  including  candidate  events  in  the  analysis: 

(1)  the  development  must  be  important 
to  the  subject  being  evaluated,  in 
the  sense  that  the  event  should 
have  some  impact  on  an  action  or 

a decision  that  is  being  contem- 
plated. 

(2)  only  those  important  developments 
whose  outcome  is  uncertain  (that 
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is,  those  that  may  occur  as 
opposed  to  those  that  almost 
certainly  will  or  will  not  occur) 
should  be  included  in  the  set 
being  analyzed. 

(3)  if  the  event  set  is  to  be  highly 
limited  it  is  also  desirable  for 
the  items  to  be  causally  re- 
lated to  each  other  in  some 
way.  (Ref.  35,  p.  345)12 

(4)  Developments  or  events  as  referred 
to  here  denote  specific  items 
which  may  occur  in  the  future. 

They  are  'specific'  in  the 

sense  that  their  occurrence 
or  non-occurrence  can  be 
rigorously  determined. 

(Ref.  27 , p.  50)  . 

Mitchell  and  Tydeman  propose  several  rules  for  event 
formulation.  Their  purpose  is  to  reduce  possible  respondent 
misinterpretation  and  thereby  reduce  response  variation. 
These  rules  will  be  followed  in  the  application  section  of 
this  study. 

(a)  Except  where  the  cause  is  part 
of  the  event  to  be  studied, 
events  should  not  be  stated  in 
the  form  'A  occurs  because  of  B' . 

(b)  Events  should  be  single  rather 
than  multiple,  i.e.,  'A  occurs' 
rather  than  'A,B,  and  C occur'. 

(c)  Events  should,  where  possible, 
be  definite  and  specific  rather 
than  vague  and  general,  e.g.,' 


12 

Enzer  develops  the  requirement  of  being  "causally  related" 
through  the  concept  of  "coupling" . He  describes  three  types 
of  event  coupling,  "totally  uncoupled",  "coupled",  and  totally 
included".  Of  these,  totally  uncoupled  events  are  felt  to 
be  of  little  importance  in  the  analysis,  coupled  events 
should  be  analyzed,  and  included  events  may  be  analyzed  if 
their  impacts  are  not  estimated  more  than  once  [Ref.  27,  p.  52J . 
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'Costs  will  increase  more  than 
20%  in  real  terms'  rather  than 
'Costs  will  increase  signific- 
antly'. (Ref.  29,  p.  136]. 

Formulation  rules  are  necessary  to  insure  that  "the  group 
involved  reach  agreement  and  understanding  on  the  specifica- 
tion and  wording  of  the  event  set."  [Ref.  36,  p.  333]. 

Although  it  is  desirable  that  the  event  set  completely 
define  the  problem  under  consideration,  this  is  rarely  possi- 
ble. The  uncertainties  in  the  problem  environment,  and  the 
complex  issues  involved  could  necessitate  specifying  many 
events.  Since  the  possible  first  order  interactions  between 
events  increases  as  the  square  of  the  number  of  events  speci- 
fied, estimating  their  interactions  would  quickly  become 

extremely  time  consuming  [Ref.  34],  Therefore,  completeness 

13 

is  often  obtained  by  fiat.  Since  it  is  almost  impossible 
to  specify  an  event  set  which  includes  every  conceivable 
development  most  analysts  limit  the  event  set  to  a manage- 
able size  and  attempt  to  include  the  most  important  or  signi- 
ficant events.  In  such  cases,  events  to  be  included  in  the 
analysis  are  selected  from  a large  list  of  "candidate*  events 
on  the  basis  of  subjective  importance,  desirability,  or 
impact  ratings.  These  procedures  will  be  described  in  the 
next  chapter. 


i 


^^This  limitation's  implications  have  been  examined  in 
several  cross-impact  approaches.  The  developers  (Enzer,  Turoff, 
and  Mitchell  emd  Tydeman)  suggest  different  methods  to  iden- 
tify the  most  important  events  and  to  retain  the  most  sensi- 
tive or  influential  ones  for  analysis.  In  this  manner,  it  is 
felt  that  the  most  serious  aspects  of  a problem  are  considered' 
Furthermore,  it  is  always  possible  to  expand  the  event  set 
as  circumst^mces  dictate.  Thus,  obtaining  a complete  event 
set  by  "fiat"  is  felt  to  be  the  only  practicable  method, 
and  does  not  impose  severe  limitations. 


2.  Initial  Probabilities 


Once  the  event  set  is  established  and  understood  by 
the  respondents,  the  next  procedure  obtains  estimates  for 
each  event's  "initial"  likelihood  of  occurrence.  These  esti- 
mates are  variously  termed  individual,  initial  or  marginal 
14 

probabilities.  All  refer  to  a particular  event's  likeli- 
hood of  occurring  by  some  specified  date. 

As  with  event  specification,  Enzer  suggests  that 
these  estimates  should  be  developed  from  an  environment  which 
might  evolve  naturally  from  the  current  situation  [Ref.  35]. 
This  interpretation  recognizes  that  developments  are  typically 
seen  in  their  total  environment,  and  not  in  isolation. 

This  viewpoint  poses  a conceptual  problem  with  cross- 
impact analysis  which  must  be  resolved  before  the  technique 
can  be  operationalized.  If  events  are  viewed  in  their  total 
environment  when  their  initial  probabilities  are  estimated, 
do  these  estimates  include  a consideration  of  the  occurrence 
of  other  likely  events  in  the  set?  In  other  words,  in  esti- 
mating the  initial  probability  that  event  A will  occur,  do 
respondents  implicitly  consider  the  impact  on  A of  another 
event  (say  B)  if  the  other  event  also  occurs?  Each  approach 
to  cross-impact  analysis  must  deal  with  this  problem,  and 


14 

Different  terms  are  applied  by  the  various  cross-impact 
analysis  approaches  to  describe  the  technique's  basic 
concepts.  In  this  study  these  will  be  identified  as  they 
occur,  and  thereafter  the  most  descriptive,  or  logically 
consistent  term  will  be  used. 


it  will  be  considered  more  fully  while  examining  specific 
approaches. 


A reasonable  response  to  this  problem  appears  to 
be  that  estimators  do  take  the  interrelationships  among 
events  into  consideration,  but  only  incompletely  and  not 
systematically.  Therefore,  some  modification  is  necessary 
to  obtain  "consistent"  estimates  [Ref.  34,  p.  342]. 

3 . Cross-Impacts 

The  next  process  obtains  "A  qualitative  description 
of  the  effect  of  the  occurrence  (or  non-occurrence)  of  one 
event  on  the  likelihood  of  occurrence  of  the  other  events 
in  the  set."  (Ref.  27,  p.  51].  These  are  subjective  esti- 
mates concerning  how  an  event's  occurrence  would  change  the 
probability  of  every  other  event's  occurrence.  These  are 
binary  estimates,  dealing  with  the  interactions  between  two 
events  only. 

In  eliciting  these  estimates,  respondents  are  told 
to  assume  certain  occurrence  (or  non-occurrence)  for  an 
event  and  then  assess  its  impact  on  each  remaining  events . 
Rules  for  event  impact  assessment  state  that  the  events  are 
assumed  to  happen  only  once  (i.e.,  non-repetitive) , that 
they  do  not  have  to  happen  at  all,  and  that  they  do  not 
occur  simultaneously  [Ref.  36]. 

These  estimates  have  been  termed  "conditional"  or 
"causal"  probabilities.  This  conflict  in  terms  stems  from 
another  conceptual  problem  in  cross-impact  analysis,  relating 
directly  to  the  problem  of  obtaining  consistent  estimates. 
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The  basic  problem  is  determining  how  the  cross-impacts 
should  be  interpreted.  One  interpretation  is  that  the  esti- 
[ mates  are  conditional  probabilities  in  the  "classical" 

I sense  (i.e.,  the  probability  of  event  A given  event  B has 

occurred,  P{A|b)),  or  are  used  to  derive  conditional  proba- 
I bilities  [Ref.  37] . Accepting  this  interpretation  permits 

I using  Bayesian  probability  relationships  to  test  estimate 

consistency.  Several  developers  have  derived  conditions  which 
are  intended  to  insure  consistency  and  can  be  used  to  com- 
pute probability  bounds  on  subsequent  estimates. Another 
interpretation  is  that  the  estimates  are  causal,  time-dependent 
probabilities  to  which  classical  probability  rules  do  not 
apply  (i.e.,  the  probability  A will  occur  assuming  B occurs 
first,  P(a1b^)).  In  this  case  Bayesian  probability  rela- 
tionships have  only  limited  application  [Ref.  39] . 

A third  interpretation  is  that  the  technique  attempts 
to  obtain  subjective  estimates  for  the  correlation  coeffi- 
cients between  events.  In  this  case  inconsistency  is  not 
readily  apparent,  as  when  probability  values  exceed  unity 
or  become  negative.  Turoff  applies  this  interpretation  in 
his  cross-impact  approach  [Ref.  36] . 

This  problem  will  be  exeimined  in  detail  in  the  next 
chapter.  Initially,  however,  it  appears  that  using  classical 
probedDility  relationships  not  only  permits  testing  estimates 

^^See,  for  example,  Dalkey  and  Sarin  [Refs.  34  and  38] 
for  consistency  conditions  and  boundaries. 
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for  consistency,  but  provides  relationships  for  adjusting 
inconsistent  estimates,  and  furnishes  a model  for  their 
manipulation.  Without  such  a model,  and  recognizing  the 
lack  of  causal  knowledge  in  the  social  sciences,  the  tech- 
nique faces  problems  in  interpreting  the  subjective  esti- 
mates, and  justifying  the  manipulation  algorithms . 

4 . Computational  Procedure 

The  final  cross-impact  analysis  process  employs  a 
technique  to  analyze  estimates  and  provide  useful  informa- 
tion to  decision  makers.  As  noted  by  Dalkey,  given  the 
large  numbers  of  possible  interactions, 

even  if  a matrix  describing  the  inter- 
actions is  available  . . . the  task  of 
thinking  through  the  implications 
rapidly  gets  out  of  hand.  Some  com- 
putational aid  is  required  to  take 
account  of  the  large  number  of  inter- 
dependencies. (Ref.  34,  p.  341] . 

Frequently,  the  approach  conducts  "a  Monte  Carlo 

sampling  of  chains  of  events  in  which  the  probability  of 

an  event  in  the  chain  is  modified  by  the  cross-impact  of 

the  previously  occurring  event  in  the  chain."  (Ref.  34, 

p.  341]  . This  procedure  requires  an  algorit.hm  for  calculating 

the  cross-impact  effects  among  events.  Several  methods  of 

manipulation  and  analysis  have  been  proposed  and  are  examined 

in  the  next  chapter. 


It  also  seems  that  probability  values  imply  that  A and 
B are  discrete  events;  whereas  correlation  coefficients,  "r", 
imply  that  A and  B are  indicators  of  underlying  variables  to 
which  the  events  are  linked  conceptually  within  a theoretical 
f raunework. 
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In  the  Monte  Carlo  process,  an  event  is  randomly 
selected,  and  its  occurrence  or  non-occurrence  randomly 
decided  based  on  its  initial  probability.  The  remaining 
events'  initial  probabilities  are  adjusted  using  the  selected 
algorithm,  according  to  the  first  event’s  cross-impacts. 

At  this  point  a second  event  is  selected  and  decided,  based 
on  its  new  probability.  The  remaining  probabilities  are 
revised  and  the  process  continues  until  all  events  have  been 
decided. 

Completing  the  Monte  Carlo  procedure  is  termed  a 
run.  Conducting  numerous  runs  permits  calculating  revised 
initial  probabilities  for  each  event  based  on  the  event's 
experimental  frequency  of  occurrence.  As  an  event  is  decided 
its  occurrence  of  non-occurrence  is  recorded  (either  1 or  0 
respectively) . Revised  initial  probabilities  are  then 
determined  by  dividing  the  number  of  times  each  event 
occurred  by  the  total  number  of  runs  conducted. 

Early  research  using  the  technique  suggested  con- 
ducting more  than  one  thousand  runs  in  order  to  reach  sta- 
bility and  reduce  random  variation  [Ref.  26).  However,  during 
demonstrations  as  few  as  twenty  runs  produced  adequate  re- 
sults [Ref.  27].  Johnson  examined  the  requirements  con- 
cerning how  many  runs  were  needed  to  produce  significant 
probability  shifts  and  stabilize  inherent  random  variation 
[Ref.  40] . Using  standard  statistical  techniques  he  concluded 
that;  (1)  The  mean  probability  values  for  events  reached 
stability  within  a small  number  of  runs  (<  50) . This  implies 
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that  the  model  reached  stability,  or  a consistent  point, 
relatively  quickly.  (2)  The  standard  deviations  of  modi- 
fied probabilities  decreased  with  increased  numbers  of  runs. 
However,  beyond  fifty  runs,  the  magnitude  of  these  reduc- 
tions was  small.  This  indicates  that  stability  was  reached, 
and  that  random  variation  was  only  slightly  reduced  by 
increasing  the  numbers  of  runs.  (3)  As  more  events  were 
included  in  the  .analysis  set,  no  notable  changes  appeared 
in  the  average  standard  deviations  obtained.  This  indicates 
that  the  numbers  of  runs  does  not  have  to  increase  as  the 
number  of  events  increases.  (4)  Furthermore,  Johnson  observed 
that  while  it  is  possible  to  obtain  significant  probability 
shifts  from  a statistical  standpoint,  there  is  no  assurance 
that  such  shifts  are  significant  from  a policy  standpoint 
[Ref.  40,  p.  127-130] . 

D.  CROSS-IMPACT  ANALYSIS  SYMBOLOGY 

The  different  cross-impact  approaches,  besides  using 
various  terms  to  describe  basic  concepts,  have  also  employed 
different  symbologies  in  their  demonstrations.  This  section 
is  presented  in  an  attempt  to  establish  a consistent  symbology 
for  cross-impact  computations.  This  will  simplify  comparing 
the  different  cross-impact  approaches  in  the  ne.xt  chapter. 

1.  Event  Set  Specification 

Let  A,  B,  C,  ...  N be  events  which  could  develop  from 
present  circumstances,  which  seem  to  be  related  to  each 
other  (i.e.,  are  coupled  as  previously  described),  and  are 
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important  to  some  problem  being  considered.  These  events 
were  selected  and  formulated  in  accordance  with  the  rules 
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specified  previously.  Designating  A as  event  one  (Ej^)  , B 

as  event  two  (E2)  / etc.,  produces  a set  of  events  {E^^,  E^, 

E-j  r • • • » E } . 
j n 

2.  Estimate  Initial  Probabilities 

Each  event  (Ej^  through  E^^)  has  an  associated  initial 
probability  of  occurring  by  some  specified  time  in  the 
future  based  on  the  present  situation.  It  is  possible  for 
experts  to  estimate  this  probability  (i.e.,  if  present 
developments  continue  without  change,  the  probability  A,  (E^) , 
will  occur  by  time  T is  x) . Designate  this  probability  (of 
event  A)  as  P(E-[^)  and  similarly  for  all  events  in  the  set 
{P(Ej_),  P(E2),  PCE^),  ...,  P(E^)}.  Or  as  P^  for  i = 1,2, 
3,...,n. 

3 . Estimate  Cross- Impacts 

The  next  step  is  to  estimate  the  effect  on  the  other 
events  of  the  occurrence  of  any  one  event  (i.e.,  if  event 
B is  certain  to  occur  what  is  the  new  probability  event  A 
will  occur?).  P(Ej^|E2)  or  Pdjj)  for  i,j  = 1,2,3, ...,n 
and  j ^ i.  These  estimates  are  obtained  for  each  possible 
event  pair  (i,j)  and  designated  in  the  form  P (impacted 
event  I certain  event). 

4 . Computational  Procedure 

The  result  from  this  series  of  estimates  may  then  be 
arranged  in  an  n by  n matrix.  Since  the  impact  of  an  event 
on  itself  obviously  has  no  meaning,  the  diagonal  cell  by 
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convention  contains  the  events'  initial  probability 

2 

matrix  thus  contains  all  n required  estimates. 


FIGURE  1 


BASIC  CROSS-IMPACT  MATRIX 


the  effect  on 
this  event  is 


EVENT 


if  this 
event 


occurs 


A P(Ej_)  P(E2!Ej_)  P(E^|E^)  P(Ej^IEj^) 

B P(E^iE2)  PiE^)  P(E3]E2)  P(Ej^jE2) 
C P(Ej^  E3)  P(E2lE3)  PIE^)  P(E^iE3) 
N P(E^jEj^)  P(E2lE^)  P(E3]Ej^)  P(E^) 


Note  that  P(E3^jE2)  is  not  necessarily  equal  to  P(E2lEj^). 

The  matrix  is  then  manipulated  using  the  Monte  Carlo 
procedure  previously  described,  or  some  other  procedure  may 
be  substituted.  Using  numerous  runs  permits  computing 
revised  initial  probabilities  for  each  event  based  on  its 
frequency  of  occurrence  during  the  runs. 


P'(E„) 


# times  E occurred 
n 

I runs  conducted 


E.  OUTPUTS  AND  APPLICATIONS 

This  section  will  review  the  types  of  analyses  conducted 
using  the  technique  and  their  possible  applications.  The 
technique's  proponents  have  demonstrated  several  types  of 


analyses  using  the  data  generated  by  the  technique.  Many 
applications  claimed  for  these  outputs  seem  to  address 
effectively  the  difficulties  identified  with  problem  solving 
in  the  social  sciences.  These  claims  seem  plausable  in  the 
context  of  the  authors'  arguments,  however,  final  judgment 
on  the  technique's  utility  must  be  reserved  until  the 
claims  can  be  verified  with  data  from  experiments  or  real 
world  applications.  These  outputs  and  their  supporting 
rationale  will  be  described  below,  along  with  proposed 
applications.  Specific  procedures  for  conducting  these 
analyses  will  be  presented  in  the  next  chapter  which 
develops  the  major  cross-impact  analysis  approaches. 

One  often  cited  potential  for  the  technique  is  the 
learning  experience  it  provides.  Due  to  its  systematic 
approach,  estimators  are  forced  to  develop  and  clarify 
their  opinions  regarding  the  causal  linkages  coupling 
potential  developments  in  a field  or  several  fields  into  a 
consistent  picture  [Ref.  33] . Estimators  must  be  explicit 
when  structuring  their  responses,  and  are  forced  to  con- 
sider related  impacts  and  overall  consistency.  This  aids  in 
organizing  and  evaluating  an  individual's  views  on  a com- 
plex problem  [Ref.  36] . 

Manipulating  the  matrix  through  the  Monte  Carlo  process, 
or  other  procedures  (other  procedures  will  be  discussed 
during  the  review  of  specific  cross-impact  approaches) , pro- 
duces event  sequence  scenarios  which  indicate  the  various 
ways  in  which  a given  situation  might  evolve.  As  used  in 


cross-impact  analysis  the  term  scenario  has  a very  specific 
connotation,  A scenario  refers  to  the  event  outcome  deci- 
sion (either  occurrence  or  non-occurrenc)  made  when  the 
estimates  are  manipulated,  and  the  sequence  in  which  the 
events  are  decided.  For  a six  event  set  a possible  scenario 
would  be  depicted  in  the  following  manner: 

(1,  0,  0,  1,  1,  1,) 

where  events  1,4,5,  and  6 occurred,  and  events  2 and  3 did 
not.  In  this  study,  subsequent  discussions  of  the  cross- 
impact methodology,  of  specific  cross-impact  approaches,  and 
of  the  study's  demonstration  results  will  use  the  term 
scenario  (or,  event  sequence  scenario)  to  refer  to  these 
event  occurrence  chains. 

Scenario  generation  provides  decision-makers  a background 
against  which  policy  options  may  be  evaluated. By  speci- 
fying scenarios,  and  evaluating  them  using  subjective  event 
importance  ratings,  it  might  be  possible  to  identify  obscure 
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Sarin  discusses  scenario  generation  and  its  importance 
[Ref.  38]  . For  a set  of  n events  there  are  2*^  possible 
scenarios,  each  with  an  associated  scenario  probability, 

P(x  ).  Sarin  proposes  a method  for  computing  these  scenario 
probabilities  from  expert  estimates.  The  scenario  probabili- 
ties may  also  be  computed,  as  were  the  revised  initial  event 
probabilities,  through  the  Monte  Carlo  process.  Each  Monte 
Carlo  run  produces  one  scenario,  P(x  ) for  that  scenario  is 
computed  by  the  frequency  that  it  occurs  divided  by  the 
number  of  scenarios  generated. 


P(x„) 


♦ Xj^  produced 
# runs 


However,  this  requires  that  many  Monte  Carlo  runs  be  conducted. 
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but  important  situations  from  among  the  vast  number  of 
possibilities  [Ref.  33}.  Used  alone,  scenario  generation 
at  least  indicates  to  decision-makers  the  likely  consequences 
of  inaction.  The  scenario  output  is  the  technique's  major 
tool  in  its  most  interesting  application,  policy  analysis. 

Through  comparing  the  scenarios  (produced  by  recording 
event  sequence  outcomes  during  a Monte  Carlo  simulation)  to 
the  cross-impact  matrix  it  is  possible  to  identify  "critical" 
events  which  make  a considerable  difference  in  outcomes  [Ref. 
28].  This  causal  analysis  also  provides  "a  means  of  gaining 
insight  into  key  branch  points  and  items  on  which  current 
actions  and  policies  should  be  focused  in  order  to  increase 
the  likelihood  of  achieving  a desired  outcome."  [Ref.  33, 
p.  451. 

By  examining  the  cross- impact  matrix  with  the  scenarios 
it  is  also  possible  to  identify  possible  "side  effects"  pro- 
duced by  the  decision  under  consideration.  This  examination 
could  aid  the  decision-maker  in  two  ways.  First,  a nagging 
problem  which  faces  decision-makers  is  to  identify  additional 
problems  which  may  arise  further  down  the  line  caused  by 
their  attempts  to  respond  to  the  immediate  problem  situation. 
Determining  a decision's  possible  side  effects  may  reveal 
unanticipated  developments  which  might  stem  from  a particular 
situation  or  a particular  policy  option. 

A second  advantage  offered  by  exploring  a decision's 
side  effects  is  that  the  examination  may  indicate  less 
expensive  means  of  attaining  goals  through  investment  in 


high-payoff  areas  which  may  have  seemed  unrelated  [Ref.  26, 
p.  101].  This  advantage  is  reflected  in  attempts  to  use 
cross-impact  analysis  to  evaluate  resource  allocation  strat- 
egies. Gordon  suggested  resource  allocation  strategies  could 
be  tested  by  adjusting  the  initial  probability  estimates  for 
one  or  several  events  to  reflect  policy  decisions  in  order  to 
improve  or  inhibit  their  likelihood.  These  adjustments  are 
then  manipulated  as  was  the  original  matrix,  and  the  output 
examined  to  determine  the  policy  decision's  effect  on  the 
various  scenario  probabilities  observed  [Ref.  24].  This 
process  is  termed  sensitivity  analysis,  and  can  be  used  to 
explore  the  effects  caused  by  changes  in  problem  configuration. 

Further  cross-impact  analysis  applications  derive  from 
using  the  technique  to  model  the  relationships  operative  in 
a given  situation  [Ref.  4].  Enzer  demonstrates  that  the 
matrix  may  be  used  to  determine  an  event's  dominance  and 
sensitivity.  In  his  approach,  the  sum  of  the  absolute  values 
in  a matrix  row  can  be  interpreted  as  a measure  of  the  magni- 
tude of  the  impact  one  event’s  occurrence  has  on  the  occur- 
rence likelihood  of  all  other  events  (its  dominance)  [Ref.  27, 
p.  54].  As  a minimum,  this  identifies  the  events  which  have 
the  greatest  and  least  effect,  relatively,  on  the  situation. 
Those  events  with  the  greatest  overall  effects  are  prime 
candidates  for  intervention  through  resource  allocation. 

Event  sensitivity  is  computed  from  the  algebraic  sum 
of  the  matrix  column  values.  This  is  a measure  indicating 
how  responsive  an  event  is  to  the  other  events  in  the  set 
(Ref.  27,  p.  54].  It  reveals  events  which  may  be  susceptible 


to  change  through  secondary  intervention.  It  could  also 
indicate  those  events  which  most  readily  indicate  changes  in 
the  system,  and  hence,  should  comprise  the  components  of  a 
monitoring  mechanism. 

By  focusing  on  the  opinion  spread  registered  by  the 
experts,  Enzer  develops  an  uncertainty  measure  reflecting  the 
degree  of  estimator  uncertainty  concerning  the  cross-impacts. 
This  is  derived  for  each  event  by  determining  the  distance 
between  the  upper  quartile  and  lower  quartile  estimate  values 
[Ref.  27,  p.  541.  This  "uncertainty"  measure  also  highlights 
the  level  of  agreement  cimong  the  experts,  and  can  be  inter- 
preted as  a "confidence"  measure.  Such  information  would  be 
an  aid  in  resource  allocation,  allowing  decision-makers  to 
avoid  investing  effort  in  influencing  an  event  whose  impact 
was  highly  uncertain.  If  subjective  estimates  regarding  an 
event's  importance  and  desirability  are  obtained  with  the 
probedaility  estimates  more  elaborate  analyses  are  possible. 

These  could  aid  decision-mcUcers  in  identifying  desirable  actions 
and  areas  warranting  further  investigation  [Ref.  27,  p.  57] . 

The  cross-impact  technique  also  appears  amenable  to  mani- 
pulation with  digraph  theory,  and  other  techniques  used  in 
studies  of  individual  and  organizational  information  process- 
ing (esp.  information  theory) . Digraph  theory  uses  binary 

matrix  entries  to  trace  third-  and  fourth-level  interrelation- 

18 

ships  aunong  events.  Turoff  incorporated  information  theory 

18 

The  principles  and  methods  of  digraph  theory  are 
presented  in  Harary,  et  al.  [Ref.  42] . Axelrod  [Ref.  43] , and 
Bonham,  et  al.  [Refs.  44  and  45]  have  attempted  to  apply  digraph 
theory  to  international  relations  theory  and  problems. 


in  his  cross-impact  approach  to  relate  estimates  for 
occurrence  and  non-occurrence  [Ref.  36]. 

F.  RATIONALE  FOR  SELECTING  CROSS-IMPACT  ANALYSIS 

The  preceding  discussion  has  revealed  several  aspects  of 
cross-impact  analysis  which  make  it  attractive  as  a decision- 
aiding  methodology  for  crisis  situations.  The  technique's 
advantages  will  be  summarized  below  as  justification  for 
selecting  cross-impact  analysis  for  demonstration  and  evalu- 
ation in  this  project.  Specific  aspects  of  the  technique 
discussed  include:  (1)  its  subjective  method  of  evaluation, 

(2)  its  systematic  approach,  (3)  its  flexibility,  (4)  its 
demonstrated  application  to  policy  analysis,  and  (5)  its 
potential  utility  as  a learning  tool. 

1.  Subjective 

Using  expert's  subjective  evaluations  precludes  the 

need  for  a formal  theory  relating  crisis  elements,  national 

interests,  and  policy  options.  The  procedure  takes  into 

account  the  perceptions  and  accumulated  knowledge  of  experts 

and  decision-makers  (which  is  not  easily  transferred)  and 

uses  them  to  process  and  interpret  imprecise  information. 

It  minimizes  the  need  to  collect  precise  data  on  numerous, 

inadequately  understood  indicators,  especially  in  situations 

19 

where  time  and  stored  information  is  limited. 
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This  is  not  intended  to  suggest  that  theory  construction 
and  indicator  development  are  not  important.  As  noted  in 
Chapter  One,  these  approaches  are  required  for  system  moni- 
toring, furthermore,  they  are  necessary  to  achieve  precision 
in  the  social  sciences  as  indicated  by  Rossenau  [Ref.  46] . 

What  is  intended  is  to  establish  the  current  need  for  sub- 
jective estimates  in  quantitative  analyses  of  policy  problems. 


. Systematic  Approach 


Because  the  technique  is  systematic  in  its  approach, 
it  encourages  the  consideration  of  the  numerous  interactions 
which  could  be  overlooked  in  situations  involving  short  response 
time  and  stress.  In  addition,  it  proceeds  logically  from 
problem  identification  through  the  selection  of  policy  options. 
The  subjective  estimates  obtained  during  the  evaluation  process 
are  retained  so  the  entire  analysis  can  be  replicated  at  will, 
rather  than  being  lost  after  the  crisis  has  passed. 

Finally,  because  it  is  a systematic  procedure  cross- 
impact analysis  may  be  conducted  using  a computer.  Most 
applications  demonstrated,  especially  those  presented  by 
Mitchell  and  Tydeman,  Turoff,  and  Sarin  [Refs.  29,  36,  and 
381 » have  utilized  or  advocated  an  interactive  computer 
system  for  querying  experts.  This  capability  increases 
analysis  speed  and  flexibility,  both  of  which  are  desirable 
for  a decision-aid  in  crisis  situations. 

3.  Flexibility 

Cross-impact  analysis  is  not  constrained  by  data 
requirements  to  evaluating  only  a few  empirically  quantifiable 
variables . The  variables  used  in  the  technique  are  obtained 
from  experts'  perceptions  concerning  a specific  problem.  If, 
while  obtaining  subjective  probability  estimates,  it  becomes 
apparent  some  important  aspect  of  the  problem  has  been 
omitted  it  is  possible  to  include  it  in  the  event  set.  This 
capability  is  particularly  useful  in  analyzing  a crisis 
situation  involving  uncertainty  and  incomplete  information. 
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It  allows  the  event  set  to  be  modified  in  response  to 
changing  circumstances.  Such  "last  minute"  modifications 
are  possible  because  the  values  used  in  the  analysis  are 
subjective,  and  can  be  easily  revised  or  expanded. 

Using  subjective  judgments  also  permits  the  approach 
to  address  any  problem  for  which  an  appropriate  event  set 
can  be  specified.  This  flexibility  is  evidenced  by  the  broad 
spectrum  of  problems  to  which  the  technique  has  been  applied. 

4 . Demonstrated  Applications 

Several  cross-impact  studies  have  been  conducted  in 
situations  relevant  to  policy  analysis.  Heuer  describes  one 
such  study  in  reviewing  CIA  efforts  to  use  quantitative 
techniques  to  analyze  policy  relevant  situations  [Ref.  41]. 

The  approach  is  future  oriented  and  addresses  the  potential 
evolution  of  a current  situation.  Thus,  cross-impact  analysis 
attempts  to  be  predictive,  conducts  sensitivity  analysis,  and 
indicates  degrees  of  uncertainty.  Other  developers  have 
applied  the  methodology  to  problems  of  evaluating  transporta- 
tion alternatives,  population  trends,  and  energy  options 
[Refs.  26,  33,  and  47].  Changing  values  and  objectives  are 
implicitly  considered  by  using  subjective  estimates  or  may 
be  explicitly  stated  in  the  cross-impact  matrix.  Finally, 
once  the  estimates'  form  and  meaning  are  understood  the 
procedure  may  be  used  by  anyone.  For  these  reasons,  cross- 
impact analysis  enhances  the  process  of  analyzing  a situation 
and  reaching  a policy  decision. 
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5.  Learning  Tool 


Since  respondent  estimates  can  be  retained,  the 
technique  can  help  close  the  feedback  loop  involved  in 
assessing  the  direction  in  which  a situation  is  moving.  - This 
can  be  accomplished  by  comparing  estimates  obtained  over  time 
(?!  - T2) . If  a policy  has  been  implemented  between  and  T2» 
then  the  comparison  tests  the  policy's  success. 

By  monitoring  the  emergence  of  cross-impact  matrices 
over  time,  it  might  be  possible  to  gain  insight  into  how 
organizations  structure  their  knowledge  about  the  environment. 
Thus,  a series  of  matrices,  tadcen  from  a particular  situation, 
might  reveal  some  mechanics  of  organizational  information 
processing. 

G.  EVALUATION  CRITERIA 

The  preceding  overview  highlighted  cross-impact  analysis' 
potential  applicability  as  a policy  tool.  However,  the 
various  conceptual  problems  identified  here  and  by  the  tech- 
nique's developers  have  produced  several  alternative  cross- 
impact approaches,  each  advancing  different  methods  to  deal 
with  the  technique's  conceptual  problems,  and  leading  to 
confusion  regarding  which  variation  of  the  technique  is  the 
best  to  operationalize. 

Due  to  the  confusion,  selecting  a variation  for  extension 
as  a crisis  decision-aid  should  be  approached  systematically 
using  consistent  evaluation  criteria.  These  criteria  should 
address  each  variation's  primary  components,  including  its 
assumptions,  underlying  rationale,  and  implications.  Specific 
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implementation  problems  should  be  evaluated  by  examiining 
information  requirements,  computational  procedures,  and 
methodological  approach.  The  goal  is  to  select  a practicable 
approach  and  develop  it  into  a crisis  decision-aid. 

In  developing  consistent  evaluation  criteria,  several 
considerations  must  be  kept  in  mind.  These  stem  from  the 
characteristics  defining  crisis,  and  this  project's  time  and 
computational  resource  limitations.  Based  on  these  considera- 
tions, discussed  in  the  first  chapter,  the  following  criteria 
were  formulated  for  comparing  the  various  cross-impact  approaches, 

1.  Is  the  event  formulation  scope  adequate?  This 
refers  to  the  subject  area  addressed  by  the  event  set.  In  the 
intended  analysis,  events  must  cover  such  concepts  as  "national 

-interests"  threatened,  and  specific  "policy  options"  available. 

2.  Can  events  be  rated  on  importance/desirability? 


The  need  for  such  subjective  estimates  in  sensitivity  analysis 
and  event  set  selection  has  been  indicated.  This  refers  to 
whether  or  not  the  specific  technique  includes  procedures  for 
obtaining  and  using  such  estimates. 

3 .  What  types  of  analysis  can  be  performed  using  the 
technique? 


4 . Is  the  technique  computer  adaptable? 

5 . Are  the  estimated  relationships  clearly  formulated? 
This  refers  to  whether  or  not  the  procedure  used  to  obtain 
estimates,  and  the  questions  asked,  are  understood  by  the 
respondents.  Are  the  respondents  asked  for  estimates  they 


can  reasonably  be  expected  to  supply? 
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6 . Are  the  estimates  consistent,  or  can  consistency 
be  obtained?  This  refers  to  the  need  for  estimates  which  can 
be  manipulated  using  a systematic,  rigorous,  methodology. 

7.  What  algorithm  is  used  to  manipulate  the  estimates? 
This  refers  to  the  procedure  by  which  the  cross-impacts  are 
"taken  into  account"  [Ref.  34,  p.  342],  and  how  it  is  structured. 

8.  Is  the  procedure  logical/ j ustifiable?  This 
refers  to  the  assumptions  made  in  adopting  the  algorithm,  are 
they  reasonable  and  what  limitations  do  they  impose? 

9.  How  is  consistency  obtained?  This  refers  to  the 
computational  procedure  for  checking  consistency  and  revising 
estimates  into  a consistent  set. 

10.  How  are  scenarios  generated?  This  refers  to  the 
procedure  used  to  develop  event  sequence  scenarios,  how  complex 
it  is  and  whether  or  not  it  provides  a ranking  system. 

11 . How  many  estimates  does  the  technique  require? 

12.  What  types  of  computation  are  performed?  This 
refers  to  the  complexity  of  the  calculations.  Are  they 
algebraic,  quadratic,  differentials,  or  linear  programs? 

13 . How  many  terms  are  involved  in  each  formula? 

14 . How  many  separate  calculations  are  required? 

This  refers  to  what  computational  steps  are  required  for 
estimate  manipulation,  checking  consistency,  scenario 
generation  and  analysis. 

These  last  four  criteria  are  proposed  in  order  to 
identify  an  approach  which  can  be  demonstrated  using  limited 
computational  facilities.  This  study  will  replicate  the 


selected  approach  manually,  therefore,  the  number  and 
complexity  of  estimates  and  computations  must  be  kept  to 
a manageable  number. 

H.  CONCLUSION 

This  chapter  has  presented  a broad  overview  of  cross- 
impact analysis,  identifying  its  basic  concepts  and  potential 
applications.  Since  there  are  a number  of  alternative 
approaches  to  implementing  the  methodology  a set  of  evaluation 
criteria  was  developed  to  use  in  selecting  an  approach  for 
operationalization.  The  next  chapter  will  review  the  major 
cross-impact  approaches,  select  one  for  demonstration,  and 
extend  it  to  deal  with  a crisis  situation. 

Subsequent  chapters  will  discuss  the  implementation 
procedure  used,  identify  the  problems  encountered  in  opera- 
tionalizing the  selected  approach,  and  present  the  demon- 
stration results.  In  the  final  chapter,  these  results  and 
the  implementation  experience  will  be  used  to  evaluate  the 
selected  cross-impact  approach's  potential  utility  as  a 
crisis  decision-aid. 
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III.  THE  MAJOR  CROSS- IMPACT  APPROACHES 

A.  INTRODUCTION 

This  chapter  will  identify  and  evaluate  the  major  appro- 
aches to  cross-impact  analysis  usimj  the  evaluation  criteria 
developed  in  the  preceding  chapter.  Four  major  approaches 
will  be  described  and  discussed  in  terms  of  their  basic 
assumptions,  procedures  and  outputs.  In  addition,  three 
ancillary  approaches  which  were  felt  to  be  too  complex  for 
demonstration  will  be  reviewed  briefly.  The  four  major 
approaches  will  then  be  compared  using  the  evaluation  cri- 
teria, and  one  will  be  selected  for  demonstration  and  evalua- 
tion as  a possible  crisis  decision-aid.  The  selection  pro- 
cess will  also  highlight  possible  problems  in  operationalizing 
the  selected  approach. 

Demonstrating  the  selected  approach  in  a hypothetical 
policy  relevant, crisis  situation  may  require  that  it  be 
extended  to  address  specific  problems  inherent  in  svich 
situations.  These  problems  include  short  respojtse  time, 
and  evaluating  specific  policy  options.  Possible  extensions 
to  cross-impact  analysis  will  be  developed  once  an  approach 
has  been  selected. 

B.  THE  BASIC  APPROACH  (GORDON) 

The  concept  of  cross-impact  analysis  was  introduced  by 
Gordon  and  Hayward  in  1968  [Ref.  26].  As  previously  noted, 
it  was  a partial  response  to  one  of  the  Delphi  criticisms 
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iRef . 24] . Its  first  application  was  in  a forecasting 
game,  developed  for  the  Kaiser  Corporation,  which  tested 
resource  allocation  strategies  [Ref.  24). 

1 . Assumptions 

Cross-impact  analysis  is  based  on  two  assumptions. 

In  response  to  the  Delphi  criticism,  the  developers  hypothe- 
sized that  most  events  are  connected  in  some  way.  Further, 
they  felt  that  it  is  possible  to  estimate  these  connections 
subjectively  as  to  mode  (enhancing  or  inhibiting) , strength 
(amount  of  influence) , and  time  lag  (length  of  time  required 
for  effect  to  be  felt)  (Ref.  26,  p.  101-4].^^ 

Given  that  linkages  existed  and  could  be  estimated, 
the  developers  next  addressed  how  the  impacts  wor)<ed,  or 
how  one  event's  occurrence  , changed  a second  event's 

(E^)  , occurrence  probability.  They  assumed  each  event 
had  some  "individual"  (inital)  probability  of  occurring 
^n' ^^i^ ' ^ specific  date.  The  change  in  probability 

P'j^,(P'^),  caused  by  another  event's  occurrence  was  assumed 
to  be  a function  of  mode,  strength,  and  time  lag.  However, 
an  event's  failure  to  occur  (non-occurrence)  was  assumed  to 
have  no  effect  on  the  remaining  probabilities. 


20 

These  relationships  are  more  complex  than  indicated 
here.  Gordon  realized  that  the  "connections  are  subtle 
and  vary  widely  in  strength,  [sic]  Composed  of  at  least  two 
major  elements;  probabilities  of  feasibility  (P„)  and  pursuit 
(Pp) [Ref.  24,  p.  139) . 
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stated  mathematically,  they  assumed:  [Ref.  26, 


p.  104). 


f (P^,M,S,tj,t) 


where : 


function 
estimated  Mode 

time  in  the  future  occurs 
estimated  strength 

time  in  the  future  for  which  probabilities 
are  being  estimated 


The  relationship  between  events  was  assumed  to  be  quadratic 
[Ref.  26,  p.  105], 


aP^  + bP_^  + c 


By  assuming  known  end  conditions: 


when  P^  = 0 


Pi  » 1 


then  P ' ^ 


and  if  tj  = t (no  time  lag) 


And  assuming  the  mode,  strength,  and  time  effects  are 


linear  [Ref.  26,  p.  106] 


t - t , 


where 


k » ti  (Mode) 

S = number  between  0 and  1 (smaller  representing 
weaker  effect) 

Therefore,  in  the  inhibiting  case, 

-1  < a < 0 


and  enchancing 


0 < a < 1 


this  produces: 


U - t . 2 '-i 

PV  = kS  — ^ + [1-kS  - 

2.  Procedure 

This  equation  was  demonstrated  in  two  applications, 
a historical  context  and  a futures  context.  In  these 
demonstrations,  the  estimates  were  made  by  the  experimenters. 
The  historical  problem  exeuained  the  Minuteman  deployment 
decision,  the  future  problem  examined  transportation  develop- 
ments in  light  of  technological  advances  and  changing 
societal  attitudes  (Ref.  26] . 

In  both  cases  the  procedure  began  by  specifying 
relevant  events  and  developments  to  be  examined.  These 
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events  were  obtained  through  literature  searches,  expert 
interviews,  and  the  analysts'  own  opinions. 


The  next  step  was  to  estimate  the  initial  proba- 
bilities and  occurrence  dates  for  each  event.  An  event 
matrix  was  then  constructed,  and  the  potential  interactions 
between  each  event  and  every  other  event  were  estimated. 

These  estimates  assessed  both  the  interaction's  mode 
and  its  strength.  They  were  made  by  assuming  one  event  was 
certain  to  occur,  and  then  predicting  its  impact  on  each 
remaining  event.  These  values  ranged  from  -1.0  to  +1.0, 
and  made  up  the  matrix  body.  Upon  reviewing  the  individual 
event  pair  interactions  certain  "predecessor-successor" 
relationships  were  apparent  to  the  experimenters.  In  the 

subsequent  manipulation,  events  which  had  to  be  predecessors 

21 

were  evaluated  first. 

i 

A computer  was  programmed  with  the  quadratic 
relationship  amd  the  "Monte  Carlo"  procedure  used  to  manipu- 
late the  matrix.  If  an  event  was  decided  to  occur,  its 
impact  was  calculated  from  its  cross-impacts  with  each 
other  event  using  the  quadratic  equation,  and  the  remaining 
events'  initial  probabilities  were  adjusted  accordingly. 

If  non-occurrence  was  decided  then  the  remaining  probabilities 

E 

t 


Ai 


if 
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In  the  matrix,  these  relationships  were  indicated  by 
appending  a second  digit  to  the  interaction  values.  (0  = 
immaterial,  1 = likely  predecessor,  2 = necessary  predecessor) 
[Ref.  26,  p.  107] . 
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were  not  changed.  The  computer  run  was  conducted  one  thou- 

I 

sand  times  so  that  "revised"  probeibilities  could  be  computed 
for  each  event  based  on  its  occurrence  frequency  [Ref.  26,  ^ 

p.  108]. 

3 . Analyses  Performed  , 

( j 

Using  this  procedure,  Gordon  and  Hayward  demonstrated  f.j 

that  four  potentially  useful  analytical  methods  could  be 

developed  from  the  initial  matrix.  These  were:  (1)  final 

probability  rankings,  (2)  probability  shifts,  (3)  tracing 

causal  effects,  and  (4)  scenario  sensitivity  analysis. 

Ranking  by  final  probability  produced  a possible 

event  sequence  outcome  scenario.  In  the  example  this  was 

felt  to  be  logically  consistent  (plausable) . However,  the 

developers  realized  it  was  no  more  consistent  than  other 

possible  scenarios.  (Alternative  scenarios  could  be  obtained 

♦ * 

by  recording  the  "decisions"  made  in  each  computer  run  using  | 

1* 

0 for  non-occurrence  and  1 for  occurrence.)  [Ref.  26,  p.  108]. 

1 

The  ranking  by  probability  shift  (difference  between 

I 

and  P'j_)  was  used  to  identify  the  events  most  affected 
by  the  suspected  interactions.  In  other  words:  "the  item  | 

[event]  which  had  the  highest  probability  shift  could  be 

i 

I 

expected  to  be  the  one  most  influenced  by  the  external  i 

i 

events  depicted  by  the  remainder  of  the  list."  [Ref.  26, 

p.  108]. 

In  their  forecasting  analysis,  the  developers 
attempted  to  use  the  matrix  to  trace  through  the  reasons 
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for  an  event's  resulting  probability  shift.  For  example 

they  concluded  that: 

Within  the  groups  of  items  [events] 
describing  future  transportation 
modes,  fast  sub-surface  trains 
showed  the  highest  probability 
gain,  suggesting  that  changing 
customer  preferences,  social 
customs,  and  technological 
innovations  might  favor  this  role. 

[Ref.  26,  p.  113]. 

In  essence,  this  is  an  elaborate  way  to  state  the 
probability  shift  conclusion,  and  later  cross-impact  developers 
would  attempt  to  be  more  explicit  in  tracing  causal  effects. 

The  developers  also  used  their  model  to  examine  the 
potential  effects  of  conscious  policy  decisions.  Policy 
decisions  were  interpreted  as  decisions  to  allocate  re- 
sources to  making  a particular  event  or  set  of  events  more 
likely.  Such  a decision  was  reflected  by  arbitrarily  raising 
the  initial  probabilities  for  those  events.  The  policy 
decision's  results  were  then  indicated  by  manipulating  the 
matrix  using  the  new  initial  values  and  comparing  the 
effects  produced,  on  the  event  sequence  scenarios,  with  the 
initial  run  [Ref.  26,  p.  115]. 

4.  Critique 

Gordon  and  Hayward  recognized  that  their  approach 
had  several  weaknesses.  These  would  be  expounded  upon  by 
later  reviewers.  The  quadratic  relationship  had  been  selected 
due  to  its  intuitive  appeal  and  for  convenience,  however, 
the  authors  were  not  insistent  that  it  be  used.  They 
suggested  that  other  possible  forms  be  examined.  Gordon 
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and  Hayward  also  recognized  that  there  was  little  justifi- 
cation for  assuming  mode,  strength,  and  time  lag  acted 
linearly.  A logistic  function  was  recommended  as  being  more 
appropriate.  The  assumed  non-effect  of  non-occurrence  was 
felt  to  be  an  oversimplification  which  could  be  easily  over- 
come. Finally,  in  future  applications  the  developers  pro- 
posed that  estimates  be  obtained  by  a consensus-building 
technique  such  as  Delphi  (Ref.  26,  p.  115]. 

One  of  Gordon  and  Hayward's  critics,  Enzer,  criticized 
the  quadratic  adjustment  assumption  on  the  basis  that  it 
arbitrarily  restricted  the  possible  domain  of  change  in 
P'^.  The  greatest  absolute  change  in  could  occur  only 
when  P^  = 50%.  The  quadratic  did  not  allow; 

. . . the  occurrence  of  one  event  to 
have  a large  positive  absolute 
impact  on  an  event  whose  initial 
probability  is  very  low,  or  a large 
negative  absolute  impact  on  an 
event  whose  initial  probability 
is  very  high.  [Ref.  35,  p.  352] . 

He  notes  that  although  this  appears  to  violate  intuitive 

thinking,  it  did  prevent  PV  from  reaching  unity  or  becoming 

negative. 

Enzer  also  explained  the  limitations  produced  by 
ignoring  an  event's  non-occurrence.  He  felt  that  failing 
to  consider  non-occurrence  leads  to  ambiguity  in  defining 
the  initial  probabilities.  This  is  because  any  cross-impact 
effects  which  may  have  been  implicitly  included  in  estimating 
the  initial  probabilities  are  not  removed.  However,  Enzer 
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noted  that  it  may  be  conceptually  difficult  to  estimate 
the  effect  of  non-occurrence  (Ref.  27,  p.  59]. 

In  their  estimation  of  the  interrelationships,  Enzer 
felt  that  Gordon  and  Hayward  had  not  adequately  defined  the 
cross-impact  factors  [Ref.  27,  p.  53].  This  problem  relates 
to  how  estimator  responses  should  be  interpreted,  as  dis- 
cussed in  the  preceding  chapter.  Two  other  critics,  Mitchell 
and  Tydeman,  indicated  this  problem's  implications  when  they 
observed  that  Gordon  and  Harward's  "...procedure  contains 
no  mechanism  to  ensure  that  basic  pairwise  probability 
relationships  remained  intact."  [Ref.  29,  p.  134]. 

The  basic  approach  had  other  prcolems  associated 
with  its  use.  Dalkey  points  out  that  relative  likeli- 
hoods for  individual  event  sequences  (scenarios)  could 
not  be  assessed  directly.  This  was  because  only  binary 
interactions  were  considered  [Ref.  34]. 

To  make  scenario  computation 
useful  it  will  be  necessary 
either  to  obtain  estimates  of 
higher  order  interactions  or 
find  a more  logically  correct 
assumption  concerning  them 
[Ref.  34,  p.  350]  . 

Another  problem  noted  by  Mitchell  and  Tydemand  was  that  the 
approach  lacked  an  uncertainty  measure  with  which  to  evaluate 
the  expert’s  opinions  [Ref.  29,  p.  134].  Also,  as  discussed 
in  Chapter  Two,  Johnson  challenged  the  assumption  that  proba- 
bility shifts  which  were  significant  from  a statistical  stand- 
point could  be  assumed  to  be  significant  from  a policy 
standpoint  [Ref.  40]. 
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C.  MODIFICATIONS  TO  THE  BASIC  APPROACH  (ENZER) 

In  two  successive  cross-impact  analysis  applications 
Enzer  attempted  to  overcome  some  of  the  problems  discussed 
above  [Refs.  27  and  33].  In  his  earlier  work  Enzer  retained 
the  basic  method's  approach  to  quantifying  event  interactions, 
to  using  a quadratic  manipulation,  and  to  ignoring  non- 
occurrence. The  major  innovation  was  that  he  used  the 
Delphi  technique  to  obtain  his  event  set  and  subjective 
estimates.  This  allowed  him  to  extend  the  analyses  he 
could  perform  [Ref.  27] . 

By  using  the  Delphi  technique  Enzer  was  able  to  retain 
the  estimators'  opinion  spread  for  each  specific  relationship 
examined.  Opinion  spread  is  defined  as  the  range  between 
the  upper  quartile  and  lower  quartile  response  values,  or 
one  standard  deviation  above  and  below  the  mean  value.  With 
this  information  Enzer  was  able  to  conduct  two  analyses  more 
systematically  than  Gordon  and  Hayward  had.  Enzer  demon- 
strated that  the  mean  response  values  could  be  used  to  iden- 
tify dominant  and  sensitive  events  before  their  probability 
shifts  were  determined.  In  addition,  the  estimator's  opinion 
spread  could  be  used  to  indicate  estimator  uncertainty 
concerning  the  potential  interactions. 

An  event's  dominance  was  determined  by  summing  the 
absolute  values  contained  in  a matrix  row.  Since  these 
represented  an  event's  impact  on  the  other  events,  this 
measure  was  used  to  identify  the  events  with  the  greatest 
overall  impact.  Event  sensitivity  was  measured  by  alge- 
braically summing  the  column  values.  These  values  represented 
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the  degree  to  which  all  other  events  influenced  a single 
event's  occurrence  probability.  This  measure  was  used 
to  identify  the  events  which  responded  most  readily  to 
changes  in  the  problem  situation. 

The  opinion  spread  in  a row  indicates  the  estimator's 
uncertainty  regarding  how  an  event ' s occurrence  impacts 
the  other  events.  The  opinion  spread  in  a column  indicates 
the  uncertainty  concerning  how  an  event  is  affected  by 
the  other  events  (Ref.  27,  p.  55]. 

In  addition  to  the  standard  estimates,  Enzer  elicited 
respondents'  opinions  concerning  each  event's  importance. 

He  suggested  that  these  importance  ratings,  in  conjunction 
with  the  cd30ve  measures,  can  aid  in  identifying  desirable 
events  for  resource  allocation,  and  for  selecting  events 
to  be  used  in  sensitivity  analysis.  In  this  case,  an  important 
event  which  is  insensitive  to  the  others  would  require  direct 
intervention,  sensitive  events  might  benefit  from  indirect 
action,  and  highly  uncertain  relationships  would  require 
closer  analysis  and  evaluation  before  a decision  could  be 
made  (Ref.  27,  p.  57]. 

Although  it  introduced  an  uncertainty  measure,  and 
permitted  more  exact  causal  analysis  Enzer 's  approach 
suffered  from  many  of  the  same  problems  associated  with 
the  Gordon  and  Hayward  method.  These  recurring  problems 
included:  the  use  of  the  quadratic  manipulation,  ignoring 
non-occurrence,  and  determining  how  to  interpret  the 
respondent's  estimates.  Furthermore,  the  Delphi  technique. 
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while  extending  the  analysis,  has  some  inherent  problems. 
Fischer,  et.  al.  cite  studies  which  suggest  that  the  Delphi 
technique  (where  individuals  make  estimates;  receive  anony- 
mous feedback  about  the  estimates  of  other  group  members; 
and  make  additional  estimates  based  on  the  feedback)  is  a 
poor  way  to  elicit  expert  opinion.  The  studies  reviewed  in 
Fischer,  et.  al.  indicate  that  the  best  procedure  to  elicit 
expert  opinion  uses  a slightly  different  approach.  The 
recommended  procedure  is  to  have  the  experts  make  individual 
estimates;  then,  bring  the  group  together  and  allow  the 
experts  to  discuss  their  estimates  face  to  face;  and  finally, 
have  the  experts  make  a second  set  of  individual  estimates 
[Ref.  13,  p.  55] . 

Enzer's  second  application  introduced  more  fundcimental 
changes  to  the  Gordon  and  Hayward  method.  Enzer  abandoned 
the  quadratic  manipulation,  and  introduced  the  concepts  of 
"likelihood  ratios"  and  "constant  change  of  odds"  to  handle 
the  procedure  of  manipulating  estimates.  In  this  approach, 
estimates  were  elicited  as  changes  in  probability  and  non- 
occurrence was  considered.  Enzer  also  attempted  to  resolve 
estimate  inconsistencies  [Ref.  33]. 

1 . Assumptions 

Using  an  event's  importance  rating,  it  was  possible 
for  Enzer  to  limit  the  number  of  events  to  be  included  in 
the  analysis.  Selecting  the  most  important  events  avoided 
the  necessity  to  specify  the  entire  environment,  thereby. 
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simplifying  the  analysis.  However,  this  increased  the 
complexity  of  the  estimation  process. 

Interrelationships  were  estimated  in  terms  of  the 
change  in  likelihood  of  one  event  (E^)  produced  by  another 
event's  (E^ ) occurrence.  This  was  defined  as  P(iij).  The 
effect  of  non-occurrence  could  also  be  estimated  in  the  same 
way,  and  was  defined  as  P(i|J).  If  this  second  estimate  was 
too  dififcult  for  respondents  to  provide  it  could  be  calcu- 
lated using  a standard  probability  relationship  [Ref.  33, 
p.  38]  : 


P(i)  = P{j)P(i|j)  + (1  - P{j) ) (PilJ)  ) 

These  values  were  then  used  to  compute  the  likelihood 
(cross-impact)  ratio  for  each  event  pair  [Ref.  34,  p.  350]: 

1.  . J--P(i)  , PtjU) 

‘i:  p(j)  1 - p(j i i) 

This  became  the  factor  by  which  the  initial  occurrence 
probabilities  were  multiplied  when  an  event's  occurrence 
or  non-occurrence  was  decided.  Using  this  ratio  expanded 
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the  domain  of  change  possible  for  an  event's  initial 
22 

probability. 

To  track  the  many  changes  in  probability  produced 
by  the  occurrence  or  non-occurrence  of  numerous  events, 
Enzer  introduced  the  concept  of  "constant  change  of  odds". 
He  stated: 

. . . the  judgment  concerning  the 
impact  on  Event  A of  the  occurr- 
ence of  Event  B is  independent  of 
the  exact  probability  of  occurr- 
ence of  Event  A,  and  that  when- 
ever Event  A is  preceded  by  the 
occurrence  of  Event  B the  change 
in  likelihood  of  occurrence  of 
Event  A is  known  from  the  initial 
input.  [Ref.  33,  p.  41]. 

This  means  that  once  the  interaction  between  two  events  (A 
and  B)  has  been  estimated,  its  value  is  not  changed  if 
another  event  (C)  occurs  before  B and  changes  event  A's 
initial  probability.  As  long  as  B occurs  before  A it  will 
impact  A's  current  probability  to  the  same  degree  that  it 
was  believed  to  impact  A's  initial  probability. 


22 

As  Enzer  explains  xn  his  earlier  work: 

Changes  in  probability  in  the  form  of  quanti- 
tative ratios  opens  the  domain  to  the  widest 
range  of  possibilities.  Several  of  these 
are  presented  in  Figure  6 in  a form  comparable 
with  the  previous  figure.  From  Figure  6 it 
can  be  seen  that  an  event  whose  initial  proba- 
bility was  20%  could,  if  impacted  by  an 
event  that  increased  its  likelihood  by  a 
factor  of  20,  be  raised  to  83%.  When  the 
quadratic  is  used,  the  maximum  positive  impact 
for  an  event  with  an  initial  probability  of 
20%  is  to  raise  it  to  36%.  [Ref.  27,  p.  61]. 
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In  order  to  assure  pairwise  consistency  was  maintained 
in  the  estimates  Enzer  adopted  the  following  axioms  (Ref.  33, 
p.  37]: 

1.  The  initial  probabilities  of  event  pairs 
impose  limits  on  the  values  of  the  impacted 
probabilities  for  that  pair. 

2.  For  any  event  pair,  specification  of  the 
initial  probabilities  and  one  set  of 
impacted  probabilities  (occurrence  or 
non-occurrence)  fixes  the  remaining  set. 

Thus,  the  relationship  used  to  calculate  non-occurrence  could 
also  be  used  to  check  for  consistency.  A simply  boundary 
rule  could  then  be  applied  to  resolve  inconsistencies. 

Pairwise  consistency  is  required  when  dealing  with  proba- 
bility estimates  to  insure  the  calculated  probabilities  do 
not  exceed  unity  or  become  negative.  Such  values  would  be 
meaningless  in  probabilistic  terms. 

In  applying  this  boundary  rule  Enzer  assumed  that  the 
intention  indicated  by  the  respondents'  estimates  were 
correct,  but  that  their  actual  values  were  incorrect.  The 
boundary  rule  then  makes  the  least  adjustment  required  to 
restore  consistency.  For  e.xample,  if  a value  given  for 
P(ilj)  in  conjunction  with  those  given  for  P(i)  and  P(j)  pro- 
duced a computed  value  for  P(ilJ)  which  was  <0  or  ''1,  then 
it  was  inconsistent.  A computer  would  set  the  value  of 
P(ilJ)  » 1 or  0 as  required,  and  recompute  P(ilj)  [Ref.  33, 
p.  38]. 

2 . Procedure 

Enzer 's  approach  followed  the  procedure  developed  by 
Gordon  and  Hayward.  Candidate  events  for  inclusion  in  the 
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analysis  were  obtained  from  a Delphi  exercise.  In  order 
to  limit  the  event  set  size,  only  those  events  considered 
most  important  (by  subjective  rating)  were  included. 

The  respondents  were  then  asked  to  estimate  each 
event's  initial  probability  of  occurring  by  a specified 
date,  considering  an  environment  which  might  evolve  from 
the  present  circumstances  without  conscious  human  interven- 
tion to  change  the  course  of  events.  They  were  then  asked 
to  rate  each  event's  desirability.  This  rating  included  two 
subjective  estimates;  the  desirability  of  an  event  to  the 
environmental  aspect  being  considered  (+  or  -) , and  its 
impact  on  that  environmental  aspect  should  the  event  occur 
(0  = weak  to  10  = strong)  [Ref.  33,  p.  46] . This  desira- 
bility rating  seems  to  relate  to  subjectively  assessing 
whether  an  event  enhances  or  inhibits  policy  goals,  or 
whether  an  event  influences  a situation  in  a direction 
desired  by  the  decision  unit. 

The  respondents  were  next  asked  to  assess  the  inter- 
relationships between  events.  They  provided  the  new  likeli- 
hood of  occurrence  for  each  event  given  each  other  event's 
occurrence  (and  non-occurrence  if  possible,  otherwise  this 
value  was  computed) . The  estimates  were  then  checked  for 
consistency  and  revised  as  necessary  using  the  boundary 
rule. 

Likelihood  ratios  were  then  computed  to  fill  in 
the  matrix,  and  the  estimates  manipulated  as  in  the  basic 
method,  using  a Monte  Carlo  simulation. 
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3.  Analyses  Performed 


With  this  approach  Enzer  was  able  to  perform  all 
the  analyses  permitted  by  the  basic  approach,  including; 

1.  determining  even  sensitivity  and  dominance 

2.  scenario  generation 

3.  exploring  causal  relationships 

4.  conducting  scenario  sensitivity  analysis 

In  addition  he  had  obtained  an  indication  of  respon- 
dent uncertainty  using  opinion  spread.  Rating  events  by 
desirability  allowed  selected  events  to  be  included  in 
the  sensitivity  analysis.  It  also  permitted  choosing  the 
more  desirable  events  as  subjects  for  testing  resource 
allocation  strategies. 

Furthermore,  desirability  ratings  provided  criteria 
with  which  specific  event  sequence  scenarios  could  be  eval- 
uated. Using  these  criteria,  it  was  possible  to  determine 
which  scenarios  were  the  most  desirable,  and  which  strate- 
gies produced  them. 

4 . Critique 

This  approach  overcame  several  difficulties  asso- 
ciated with  the  basic  approach,  specifically  it 

1.  considered  non-occurrence 

2.  eliminated  the  quadratic  assumption’ 

3.  demonstrated  that  expert  opinions  could 
be  used  in  analyzing  complex  situations 

4.  provided  an  uncertainty  measure  for 
evaluating  estimates 
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Nevertheless,  several  problems  still  remain. 


It  was  still  difficult  to  identify  significant 
probability  shifts,  although  the  desirability /impact  and 
importance  ratings  had  reduced  the  necessity  for  this 
determination.  The  interrelationships  considered  were 
still  only  binary,  and  even  with  desirability  ratings  it 
was  still  not  possible  to  determine  the  most  probable  scen- 
arios. Finally,  it  was  still  unclear  what  the  respondents 
meant  when  they  provided  interaction  estimates. 

This  last  problem  seems  to  have  been,  if  anything, 

confounded  by  Enzer's  attempt  to  introduce  classical  proba- 

tility  relationships.  As  Mitchell  and  Tydeman  observed: 

A fundamental  difficulty  with  cross- 
impact analysis  is  to  determine  what 
respondents  actually  mean  whn  they 
answer  the  conditional  probability 
questions  normally  posed.  Results  of 
studies  currently  in  progress  indicate 
that  (1)  participants  are  frequently 
confused  and  unsure  of  the  interpretation 
of  such  questions,  and  (2)  respondents 
often  interpret  the  question  in  terms 
of  time-dependent  conditional  probability 
statements,  naunely  'If  event  A occurred 
(first) , what  is  the  probability  of  B 
occurring?'  [Ref.  29,  p.  133]. 

In  addition,  introducing  conditional  probability 
relationships  stimulated  the  use  of  classical  probability 
theory  in  determining  estimate  consistency.  Also,  Jackson 
and  Lawton  observe  that  the  classical  probability  equation 
used  to  calculate  non-occurrence  and  test  consistency  could 
also  be  used  to  derive  half  the  matrix  entries  once  the  other 
half  had  been  specified  (Ref.  37,  p.  265]. 


Mitchell  and  Tydeman  point  out  that,  if  the  cross 


impact  estimates  are  interpreted  as  being  time-dependent, 
Enzer's  approach  may  not  be  valid. 


. . . the  standard  observation  that 
P(A)P(b1a)  = P(B)P(A|B)  beomces, 
with  the  inclusion  of  time-dependence 


P(A)P(B 


In  this  restatement  equality  is  not 
guaranteed  - in  fact,  it  would  be 
pure  coincidence  other  than  when 
events  A and  B were  independent. 
[Ref.  29,  p.  135]. 


The  consistency  question  led  Enzer  and  Alter  to 
address  using  Bayesian  probability  in  cross-impact  analysis. 
They  suggested  that  confusion  in  interpretation  was  caused 
because  the  early  approaches  [Ref.  39,  p.  231] ; 

1.  Described  future  events  with  no  real 
sense  of  time  sequence. 

2.  Used  a data  collection  technique  which 
blurred  the  distinction  between  correla- 
tion and  causality. 

They  concluded  that  conditional  probabilities  were 
not  directly  applicable  to  causal  situations  [Ref.  39,  p.  237]. 
This  conclusion  also  had  been  reached  in  1972  by  Turoff 
who  proposed  an  alternative  approach  to  cross-impact 
analysis  [Ref.  36]. 


D.  ASSUMED  CONSISTENCY  (TUROFF)  APPROACH 

Turoff  attempted  to  accomplish  several  goals  with  his 
approach.  One  was  to  develop  a method  which  could  determine 
the  causal  impact  of  potential  events  not  specified  in  the 
event  set.  He  also  wished  to  establish  some  relationship 
between  an  event's  occurrence  likelihood  and  the  effort 
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invested  in  either  promoting  or  preventing  the  event. 

In  presenting  his  approach  Turoff  acknowledged  that 

because  higher  order  interactions  are  difficult  to  esti- 

2 

mate,  "we  in  effect  ignore  terms  of  P or  greater,  hoping 
that  the  three-  or  four-way  interactions  are  small."  [Ref.  36, 
p.  326].  Nevertheless,  Turoff  did  attempt  to  derive  a 
measure  of  the  strength  of  these  higher  order  interactions. 

1 . Assumptions 

Turoff 's  approach  was  based  primarily  on  the  assump- 
tion that: 


Given  a set  of  evnets  which  may  or 
may  not  occur  over  an  interval  of 
time,  we  assume  that  an  individual 
asked  to  estimate  the  probability 
of  occurrence  of  each  event  will 
supply  a 'consistent'  set  of 
estimates  [Ref.  36,  p.  317]. 

This  assvunption  effectively  removed  the  requirement  to 


check  estimate  consistency.  Instead  estimates  were  obtained 
using  an  interactive  computer  program.  In  this  manner  the 
respondent  was  shown  the  results  produced  by  his  estimates, 
and  allowed  to  revise  them  at  will  until  he  felt  the  model 
they  produced  met  his  expectations. 

A second  assumption  in  this  approach  is  founded  on 
information  theory,  which  is  used  to  calculate  cross-impact 
factors  ) from  estimates  used  to  derive  values  for  the 
effect  of  occurrence  non-occurrence  for 

each  event.  The  assumption  was  that  "if  a correlation 
exists  between  R^^j  and  S^j  is  is  such  as  to  maximize  the 
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added  information."  [Ref.  36,  p.  324].  The  effect  of 
this  relationship  can  be  shown  simply. 


If  e is  defined  as  a quantity  close  to  0 then  [Ref.  36, 
p.  324) , 


If  Pj 

= E 

1/2 

1-E 

then 

= e 

1 

1/e 

and 

Pi  = 

Re+S  „ 

1 ® 

R+S 

R+eS 

1 

This  behaves  physically  as  one 
would  desire,  for  if  P-  is  close 
to  one,  then  a very  large  change 
in  R is  necessary  to  make  a small 
change  in  S.  Conversely,  if  P- 
is  close  to  zero,  a very  large^ 
change  in  S is  necessary  to  pro- 
duce a small  change  in  R.  Also 
when  P . * 1/2  the  relative  change 
in  R and  S is  equal.  [Ref.  36,  p.  322] . 

Finally,  a likelihood  measure  is  used  in  expressing 
the  probability  that  an  event  occurs,  called  its  occurrence 
ratio  (4»).  This  measure  is  similar  to  Enzer's  likelihood 
ratio.  Under  assumed  occurrence  or  non -occurrence  it 
ranges  from  +«  to  -“  [Ref.  36,  p.  319] . The  occurrence 
ratio  is  assumed  to  be  directly  proportional  to  a measure 
of  effort,  or  [Ref.  36,  p.  320] : 


4)P^  « 
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where 


= the  occurrence  ratio  of  event  i, 

and 

T 

E.  = the  total  amount  of  effort  expended  in 
either  bringing  about  or  preventing  "i" 


Assuming  that  estimators  are  consistent,  the  assumed 
maximization  of  added  information  allows  a difference  equa- 
tion to  be  used  for  cases  in  which  new  information  is 
received  (i.e.,  an  event's  occurrence  or  non-occurrence 
becomes  certain)  [Ref.  36,  p.  318],  Assuming  certain 

occurrence  (P.  = 1),  then,  the  approach  defines  R . . = P . , 

J ^ J ^ 

where  R^j  is  the  probability  of  the  i^^  event  given  that  j 
is  certain  to  occur.  Turoff  then  derives  a cross-impact 
factor  (occurrence  ratio)  for  occurrence  [Ref.  36,  p.  321, 
(eq.  19)]. 


Assuming  certain  non-occurrence  (P^  = 0) , then,  the  proba- 
bility  of  the  i^  event  given  that  the  j event  is  certain 
to  not  occur  is  defined  similarly,  * ^i*  ^ cross-impact 

for  non-occurrence  is  derived  [Ref.  36,  p.  321,  (eq.  21)]. 
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Mime  ijFij. 


Combining  these  two  factors  produces  [Ref.  36,  p.  321, 
(eq.  22)], 


R . . ^ S • • 
13  13 


Obtaining  values  for  all  the  C^^'s  permits  calculating 

a term  assumed  to  be  a function  of  the  unknown  variables 
not  specified  in  the  event  set  [Ref.  36,  p.  321,  (eq.  23)] 


i - 


By  assuming  the  most  favorable  case  for  event  i's  occurrence 
and  the  least  favorable  case,  two  y^'s  can  be 

computed,  y and  Y^'^.  This  permits  measuring  explicitly 
the  inaccuracy  caused  by  ignoring  higher  order  interactions 
[Ref.  37,  p.  327,  (eq.  45) ] . 


Y • “ Y • 

1 ' 1 


2.  Procedure 

The  Turoff  procedure  is  essentially  the  same  as 
the  basic  approach  until  the  point  at  which  scenarios  are 
generated.  As  in  the  basic  approach  it  is  necessary  to; 
specify  the  event  set,  obtain  initial  probability  estimates, 
and  obtain  "conditional  probability"  estimates.  These  are 
used  to  calculate  the  cross-impact  factors  as  discussed 


previously.  If  there  was  no  change  in  estimate  from  the 
initial  probability  then  C^j  = 0.  With  this  information 
it  is  possible  to  calculate  the  remaining  measures  described 
in  the  preceding  section. 

Turoff  criticized  the  Monte  Carlo  procedure  as  not 
producing  a "consistent  set  of  estimates"  because  the 
assumptions  underlying  it  imply  estimator  inconsistency 
[Ref.  36,  p.  309].  Rather  than  employing  a random  process, 
Turoff  generates  an  event  sequence  scenario  using  the 
assvunption  that  the  events  are  time- independent  in  their 
occurrence . 

The  first  step  under  this  assumption  is  to  select 
the  event  whose  initial  probability  is  closest  to  1 or  0. 

If  P(i)  is  close  to  zero  he  assumes  the  event  did  not  occur, 
if  it  is  close  to  one  it  is  assumed  to  have  occurred. 

Based  on  this  decision  the  probabilities  for  the  remaining 
events  are  recalculated  using  the  appropriate  ^ . This 
process  is  repeated  using  the  events'  new  probabilities 
until  all  events  have  been  decided  [Ref.  36,  p.  331].  This 
produces  a "unique"  scenario.  Other  possible  scenarios 
are  produced  by  changing  the  initial  probability  estimates 
and  repeating  the  above  procedure.  This  is  in  effect, 
sensitivity  analysis  as  done  in  the  other  two  approaches. 

3 . Analyses  Performed 

In  presenting  his  approach,  Turoff  demonstrates  only 
a sensitivity  analysis.  As  in  the  basic  approach,  this  is 
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accomplished  by  adjusting  one  or  more  of  the  initial  proba- 
bilities and  holding  the  remaining  probabilities  and  the 
cross-impact  factors  constant.  In  addition  to  reflecting  a 
conscious  policy  decision,  this  is  the  method  whereby  the 
estimator  determines  if  the  behavior  of  the  model  built  from 
his  responses  is  in  accordance  with  his  expectations.  If 
he  is  not  satisfied,  it  is  at  this  point  that  he  goes  back 
and  changes  his  initial  and/or  conditional  estimates  [Ref. 
36,  p.  331] . 

Although  other  types  of  analysis  were  not  demon- 
strated it  seems  most  could  be  accomplished  using  this 
approach.  Event  dominance  and  sensitivity  could  be  derived 
from  Turoff's  cross-impact  matrix  using  a procedure  similar 

to  Enzer's.  Since  Turoff's  cross-impacts  are  presented  as 

23 

a type  of  correlation  coefficient  (±  » in  range) , an 
indication  of  event  desirability  might  be  obtained  through 
simple  Slimming  of  the  positive  or  negative  effects.  Desira- 
bility analysis  would  depend  primarily  on  how  the  events 
were  formulated. 

Turoff's  matrix  also  includes  a measure  (Gamma) 
of  the  effects  of  events  not  specified  in  the  event  set 
(and  higher  order  interactions) . This  can  be  interpreted 
as  an  uncertainty  measure.  Estimator  uncertainty  could  also 


23 

The  use  of  correlation  coefficients  offers  an  alternate 
method  of  interpreting  respondent's  meanings  when  they 
answer  the  cross-impact  questions.  Its  implications  and 
possible  uses  will  be  explored  in  Chapter  Five. 


be  obtained  by  computing  the  range  of  opinion  given  by 
a group  of  estimators. 


4 . Critique 

Two  major  conceptual  problems  exist  in  Turoff's 
technique.  The  first  of  these  arises  from  the  assumption 
that  estimators  can  be  expected  to  provide  consistent 
responses.  Mitchell  and  Tydeman  reject  this  assumption 
stating : 

[Turoff]  argued  that  there  is  no  need 
for  usual  traditional  probability  rel- 
ationships to  hold  and  included  a small 
example  to  support  this  assertion.  In 
this  excunple,  he  considered  two  time 
specific  events  and  by  undertaking 
usual  conditional  probability  analysis 
concluded  that  P(A|B)  and  P(B|A)  were 
■j.  not  standard  conditional  probabilities. 

This  is  not  surprising  given  the  defin- 
tions  which  Turoff  used;  in  fact,  the 
finding  is  consistent  with  a time- 
dependent  interpretation  of  conditional 
probability.  The  result  does  not  support 
his  claim  that  traditional  probability 
relationships  are  irrelevant  to  cross- 
impact studies,  and  that  violation  of 
the  usual  probability  relationships 
can  be  justified.  The  fact  that 
people  answer  as  though  an  ordering 
cimong  events  was  implied  seems  to  be  a 
reason  for  explicitly  recognizing  this 
fact  in  the  definition  of  the  probability 
statement  rather  than  ignoring  it.  [Ref.  29, 
p.  135]. 

It  is  difficult  to  conceptualize  a situation  in 
which  an  individual  is  able  to  operationalize  all  the 
relevant  components  of  a complex  problem,  and  quantify 
their  interrelationships  to  produce  a consistent  set  of 
probability  statements  defining  his  paradigm  on  demand. 


i 

i 

i 


i 
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Furthermore,  assuming  consistency  removes  the  formal  struc- 
ture provided  the  methodology  by  classical  probability 
theory.  Lacking  an  empirical  base  in  the  inexact  sciences, 
it  seems  difficult  to  select  an  appropriate  manipulation 
technique  without  some  such  formal  theoretical  basis. 

The  second  conceptual  problem  arises  in  the  scenario 
generation  process.  By  assuming  time-independence  Turoff 
produced  one  scenario,  or  event  outcome  sequence,  but  there 
seems  to  be  little  justification  for  assuming  that  it  is 
unique.  The  assumption  that  the  most  probable  event  in  a 
situation  would  occur  first,  or  indeed  occur  at  all,  is 
debatable.  This  would  be  especially  true  in  a situation 
characterized  by  surprise.  This  indicates  that  the 
procedure  was  established  arbitrarily  and  that  several 
others  could  have  been  devised  which  would  have  functioned 
equally  well  (e.g.,  assioming  the  least  probable  event 
occurs  and  the  most  likely  does  not) . Without  some  method 
of  identifying  and  ranking  all  possible  outcomes,  production 
of  a single  scenario  seems  pointless. 

A final  criticism  of  this  technique  involves  its 
procedure.  Because  respondnets  had  to  reiterate  their 
estimates  until  satisfied  with  the  resulting  coutput,  an 
interactive  computer  system  was  required.  To  replicate 
this  reiteration  process  for  several  respondents  manually 
would  require  an  excessive  number  of  calculations  be  per- 
formed, and  much  time  be  expended  in  querying  respondents. 
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The  approach  to  be  discussed  in  the  next  section  seems  to 
overcome  these  problems. 

E.  CROSS-EVENT  ANALYSIS  (MITCHELL  AND  TYDEMAN) 

This  approach  was  developed  by  Mitchell  and  Tydeman  and 
presented  in  1978  as  a variation  to  Duperrin  and  Godet's 
SMIC-74,  which  is  discussed  in  the  following  section.  As 
has  been  shown,  Mitchell  and  Tydeman  felt  it  was  inappro- 
priate to  ignore  time  dependence  when  obtaining  estimates . 
They  felt  that  respondents  normally  interpret  the  questions 
posed  for  eliciting  the  conditional  probabilities  in  a time- 
dependent  manner.  Mitchell  and  Tydeman  attempted  to  use 
these  time -dependent  estimates  to  derive  classical  condi- 
tional probabilities  [Ref.  39] . 

Mitchell -and  Tydeman  also  recognized  that  the  elicited 
information  could  be  inconsistent  in  terms  of  classical 
probability  relationships.  To  minimize  this  problem  they 
used  average  group  responses  which  reduced  the  instances 
of  pairwise  inconsistency.  In  order  to  correct  inconsis- 
tencies which  did  manifest  themselves,  they  developed  two 
fitting  procedures,  least  squares  and  minimum  standard  devia- 
tions. These  procedures  will  be  reviewed  below,  the  authors 
also  suggested  that  Enzer's  boundary  rule  could  be  used  to 
resolve  inconsistencies  [Ref.  23,  p.  18]. 

1.  Aastimptions 

The  major  difference  between  this  and  the  other 
.ecnes  is  that  it  uses  causal  time-dependent  subjective 


probability  estimates  to  identify  first  order  interrela- 
tionships among  selected  events,  defining  them  as  classi- 
cal probability  relations.  This  approach  uses  a specific 
time  horizon,  rather  than  treating  time  as  a random  variable. 

Subjective  estimates  were  obtained  which  completely 
identified  the  eight  possible  ordered  outcomes  permitted 
by  any  event  pair.  These  outcomes  can  be  identified  explicitly 
for  any  event  pair  (i,j),  and  each  has  an  associated  unique 
probability  through  Pg)  . These  are  [Ref.  29,  p.  139]; 


Since  these  outcomes  are  exhaustive  and  mutually 
exclusive,  certain  conditions  must  apply  [Ref.  39,  p.  138]: 

^1  "^^2  ""^3  ■"^4  ■'^6  "'^7  ^^8  = ^ ^ 


+ P2  + P5  + Py  = P(i)  eq.  2 


The  authors  feel  that  respondents  can  be  expected 


to  supply  four  time-dependent  conditional  probabilities 
(Ref.  39,  p.  1391  . 


'■'llifirst'  ' 


(Pj^P,) 


(P;,^  + P2) 


(P7-HPg) 


(Pg^P^) 


eq.  4 


eq.  5 


eq.  6 


eq.  7 


A fifth  time-dependent  relationship  was  also 
presented  for  estimation  [Ref.  29,  p.  139] . 


Pdfirstl^  and  j occur)  = + P5)  eq.  8 


Solving  these  eight  relationships  as  simultaneous 
equations,  the  developers  demonstrate  that  the  probabilities 
for  an  event  pair's  ordered  outcomes  can  be  calculated  from 
the  information  provided  by  the  respondents'  estimates 
[Ref.  29,  p.  140] . 


(C3C4)  - (CgP.  - (C^P.  ) 
Y - a C,  - B C- 


(1  -C2) 


]'■ 
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If  these  last  two  equations  are  too  difficult  to 
estimate,  the  developers  derive  a method  whereby  they 
can  be  computed  [Ref.  29,  p.  141]. 


P S 

-(1  -C4)] 

. ^4  J 

• (1  - C5) ' 

P P S 

1 

n 

__  i 

P 

L s J 

^1  ^6 

u 

1 

^5 

p.  - 

1 


■ (aPg)- 

P S 

■(1  -C3)  ■ 

J 

-=3  J 

where ; 


(1  -C.)  + ; 

3 ^2 


6 = t (1  - C5)  - Cj_]  + C5  , and 


(C5C1)  + C2  - (C5C2) 


Tc^cp" 


Using  these  relationships  Mitchell  cuid  Tydeman 
calculate  the  classical  conditional  probabilities  for 
each  event  pair  [Ref.  29,  p.  140] . 


P{i|j)  = (Pj_ +P5)/(Pj_ +P2  +P5 +P7) 


P(j|i)  =*  (Pj_ +P5)/(P^ +P3 -t-Pg  +Pg) 


As  mentioned  earlier,  the  developers  derived  a 
method  to  calculate  non-occurrence  if  it  could  not  be 
estimated.  To  calculate  the  conditional  probabilities  with- 
out estimating  non-occurrence  Mitchell  and  Tydeman  adopt  the 
following  procedure.  To  obtain  conditional  probabilities 
one  of  the  relationships  needed  is; 
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•*5 

i^'4 


.-J.i 


P(i  and  j) 


P -f  P 

^1  -I- 


or 


P(i  and  j)  * P(i  and  j first)  + P(j  and  i first) 

P(a.ljfirst^  P{jfirst^  l^first)  ^^^first^ 

This  can  be  obtained  by  estimating:  ^^^first^  ^^^first^ 

or  this  can  be  approximated  by: 


P(ij)  » P(ij|i  j)  P(i)  and  similarly  for  P(jg) 


substuting : 


P(i  and  j)  » + (l-C5)C^P(j) 


which  is; 


P(i  and  j)  * P^  + P^  + E 


where  the  error  E is  small  for  independent  events,  and  more 
significant  for  highly  correlated  events  (10-20%)  [Ref.  29, 
p.  141] . 


{P,P^-P,Pg) 


(P^/Pl*P2)[(P^P,-P2P5)/(P^+P5)l  . (Pj/Pj.Pj).^^^ 
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Inconsistency  is  manifested  when  a negative  value 


is  calculated  for  any  of  the  probabilities  through  Pg. 
Standard  fitting  procedures  may  be  used  to  resolve  inconsis- 
tencies by  hypothesizing  that  there  exists  a "true"  or  con- 
sistent value  (*)  for  each  respondent's  estimate.  This 
value  is  determined  to  be  one  which  is  "as  close  as  possible" 
to  the  respondent’s  estimates  and  makes  all  Pj^,  Pg  ^ 0. 

The  developers  present  two  procedures  which  utilize  this 
hypothesis;  least  squares,  and  minimum  standard  deviations 
[Ref.  29,  p.  142] . 

The  least  squares  procedure  produces  a minimization 
program  "with  a quadratic  objective  function  and  non-linear 
constraints"  [Ref.  29,  p.  142]  . This  prograun  would  not  be 
easy  to  solve  by  hand.  Minimizing  the  absolute  standard 
deviations  produces  a moderately  sized  linear  program  with 
linear  constraints  [Ref.  29,  p.  143-4].  In  addition  this 
prograun  could  be  expamded  to  provide  for  overall  consistency 
by  including  scenario  probabilities  and  terms  for  higher 
order  interactions  [Ref.  29,  p.  145].  If  this  correction 
procedure  proves  too  lengthy,  the  authors  suggest  Enzer's 
boundary  rule  could  be  used  to  resolve  pairwise  inconsis- 
tencies. 

2.  Procedure 

To  obtain  the  events  for  analysis  the  authors  used 
a procedure  they  termed  "event  assessment" . For  each 
candidate  event,  the  respondents  provide  not  only  the 
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likelihood  that  the  event  occurs  within  a specified  time 
horizon,  but  an  estimate  of  its  significance  given  that 
it  occurs.  An  event  impact  score  is  then  calculated  as  a 
weighted  function  of  the  product  of  probability  and  signifi- 
cance. In  this  way  an  analysis  event  set  can  be  selected 
from  the  events  having  the  highest  impact  scores,  probabili- 
ties, or  significance  ratings  [Ref.  23,  p.  8]. 

The  next  step  is  to  determine  the  events ' time- 
dependent  conditional  probabilities  by  eliciting  responses 
for  questions  structured  to  provide  values  for  equations 
four  through  eight.  These  responses  are  aggregated  and  used 
to  compute  the  ordered  outcome  probabilities  through  Pg, 
which  are  used  to  compute  the  classical  conditional  proba- 
bilities. If  any  inconsistencies  appear  in  this  phase 
(any  P^^  through  Pg  < 0)  , they  are  eliminated  using  the 
boundary  rule  or  fitting  technique. 

The  next  step  is  to  generate  scenarios,  or  event 
sequence  outcomes.  For  this  procedure  Mitchell  and  Tydeman 
used  a variation  of  the  Duperrin  and  Godet  technique.  This 
technique  assumes  that  for  an  n nevent  system  there  are 
r = 2^  states  (scenarios)  . Each  state  has  a unique 

probability  [Ref.  48,  p.  305]. 

An  event's  theoretical  probability  P*(i)  is  defined 
as  its  frequency  of  occurrence  in  all  the  possible  states 
[Ref.  49,  p.  64]. 


P*(i)  » I ^ 

k=»l 


where 

► 

1 if  event  i forms  part  of 

®ik  “ ' 

0 otherwise 

A similar  definition  can  be  applied  to  the  theoretical 
conditional  probabilities  P*(ilj)  [Ref.  49,  p.  64]. 


P*(il  j) 


r 

I t(ijk)  iTj^/P*  (j)  for  all  i,j 

k»l  ' 


where 


t (ijk) 


1 when  events  i and  j form  part  of  E, 
0 otherwise 


and 


P*(i| j) 


r 

I sCijk)TTj^/l -P*(j)  for  all  i,j 
k»l 


where 


1 when  events  i and  j form  part  of  Ej^ 


s (ijk) 


0 otherwise 


Recognizing  standard  probability  constraints  [Ref.  49, 
p.  65]: 


0 _<  P*(i)  <_  1 

P*(ilj)  = P*(jli)P*(i)  = P*(ij) 
P*(i|j)P*(j)  + P*(ilJ)P*(J)  = P*(i) 

produces : 


r 

I t(ijk)7r  = P*(ij)  = P*(ji) 

k=l  ^ 


and 


P*(i|j)P*(j)  = I s(ijk)7r,  = P*(i)  - P*(ij) 

k=l  ^ 


Therefore,  given  a consistent  probability  set,  any  set  of 
values  which  satisfy: 


r 

P*(i)  = I 6., IT  for  i = 1,2, ...,n 

k=l  ^ 


r 

P*  (ij)  =»  I t(ijk)  for  i = 1,2, .. . ,n-l;  i < j ^ n 


1 


for  all  k 


r 


1 

k-1 


’^k 


are  possible  solutions  to  the  scenario  probabilities 
[Ref.  49,  p.  65]  . 

Instead  of  the  Duperrin  and  Godet  quadratic  objec- 
tive function,  Mitchell  and  Tydeman  solve  a linear  equation 
in  terms  of  consistent  probabilities  to  obtain  a unique 
solution.  This  program  is  [Ref.  49,  p.  65-6] : 


n 2 " - - 

minimize  Z » I [P(i|j)P(j)  -P*(ij)]  + [[P(i|j)P(j)  -P*(i) 

ij  ij 

+ P* (ij ) ] ^ 


n n n 

subject  to:  ^ P*(i)  - I J p*(ij)  ^ 1 

i=*l  i-1  j*i+l 


P*(ij) 


P*(i) 

P*{i) 

P*(j) 


^ 0 all  j > i 
< 1 all  i 


> P*(ij) 


all 


> P*(ij) 


j > i 


3.  Analyses  Performed 

The  cross-event  approach  uses  a different  method  to 
determine  event  dominance  and  sensitivity.  An  event's 
dominance  may  be  estimated  by  summing,  for  all  other  events. 
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the  absolute  deviation  of  the  conditional  probability, 
given  that  the  event  occurred  first,  minus  the  event's 
original  probability.  This  value  can  be  standardized, 
or  an  "elasticity"  measure  adopted  to  reflect  event  likeli- 
hood. An  event's  sensitivity  can  be  computed  in  a similar 
fashion  [Ref.  23,  p.  12]. 

Although  it  was  not  demonstrated,  causal  relation- 
ships could  be  traced  as  in  the  other  approaches  by  filling 
in  the  cross-event  matrix.  This  would  be  useful  in  an 
analysis  Mitchell  and  Tydeman  term  "system  impact  analysis" 
which  evaluates  the  possible  event  sequence  scenarios ' 
worth. 

System  impact  analysis  used  additional  subjective 
ratings  concerning  an  event  sequence  scenario's  expected 
impact  (magnitude  and  direction)  on  specified  system  com- 
ponents and  the  "expected  consequences  for  system  outcomes 
(performance  measures)"  (i.e.,  desirability)  [Ref.  23, 
p.  14] . In  this  manner  a different  set  of  possible  response 
actions  may  be  identified  for  each  scenario. 

The  final  analysis  they  suggest  is  "strategy 
evaluation".  This  procedure  relates  a set  of  feasible 
potential  strategies  (response  actions  as  determined  using 
system  impact  amalysis)  to  each  event  sequence  scenario 
(the  most  probable  ones) , and  uses  a set  of  objective 
criteria  (obtained  from  the  decision-makers)  to  evaluate  the 
strategies  for  each  possible  situation  [Ref.  23,  p.  15]. 


4.  Critique 

A major  advantage  in  this  approach  is  that  it 
eliminates  the  difficulty  in  interpreting  estimator  responses. 
Estimators  are  asked  for  time-dependent  relationships  which 
are  then  used  to  calculate  classical  conditional  probabili- 
ties. Furthermore,  if  non-occurrence  is  too  difficult  for 
respondents  to  conceptualize,  it  can  be  calculated, 
introducing  a small  error. 

The  capability  to  compute  the  conditional  proba- 
bilities results  in  the  further  advantage  of  providing  a 
logical  background  for  the  analysis.  Classical  probability 
relationships,  it  is  suggested,  can  be  used  to  insure  con- 
sistency and  resolve  inconsistencies  without  requiring 
respondents  to  re-estimate.  In  addition,  the  procedure 
develops  a method  for  calculating  and  ranking  event 
sequence  outcome  probabilities. 

With  the  scenario  ranking,  introducing  subjective 

estimates  concerning  system  impact  and  objective  measures 

for  comparison  extends  the  types  of  decision-aiding  analysis 

which  can  be  performed.  With  the  Mitchell  and  Tydeman  method 

it  is  apparently  possible  to  address  questions  relating  to 

the  event  sequence  scenarios'  desirability  and  strategy 

effectiveness.  However,  in  system  impact  analysis: 

Problems  still  face  the  analyst,  for 
ex2unple,  eliciting  an  exhaustive  objective 
set,  ensuring  mutual  independence  of 
objectives,  devising  procedures  for 
measuring  outcomes,  determining  consis- 
tent weightings  for  the  objectives 
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and  specifying  procedures  for  aggre- 
gation. [Ref.  23,  p.  15] . 

These  problems  seem  even  more  complex  for  decisions  involving 
national  objectives  and  foreign  policy  options. 

A final  difficulty  lies  not  in  the  conceptual 
realm,  but  in  the  method's  implementation.  Generating  the 
scenario  probabilities  with  the  linear  equation  for  a large 
event  set  would  be  very  difficult  without  access  to  a 
computer.  In  order  to  replicate  this  process  manually  it 
appears  necessary  to  severely  limit  the  event  set  to  be 
analyzed. 

F.  OTHER  APPROACHES 

Besides  the  approaches  to  cross-impact  analysis  dis- 
cussed so  far,  several  additional  techniques  have  been  pro- 
posed. These  techniques  were  concerned  primarily  with 
assessing  the  relative  rankings  among  the  event  sequence 
scenarios.  They  also  introduced  alternative  methods  to 
insure  estimate  consistency,  and  to  resolve  inconsistencies. 

These  techniques  took  significantly  different  approaches 
towards  resolving  these  problems  than  the  techniques  just 
reviewed.  However,  each  approach  requires  a computer  pack- 
age. Thus  they  were  evaluated  as  being  inappropriate  for 
demonstration  and  analysis  in  the  present  project.  It  was 
felt  to  be  impractical  to  replicate  their  procedures  for 
confirming  consistency,  establishing  probability  bounds 
and  revising  estimates  using  manual  methods.  However,  these 
approaches  have  merit  in  presenting  promising  alternatives 
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to  cross-impact  analysis  and  will  be  discussed  briefly. 

This  discussion  is  offered  in  the  interest  of  complete- 
ness, and  to  justify  the  conclusion  to  omit  them  from 
evaluation. 

1.  SMIC-74  (Duperrin  and  Godet) 

Duperrin  and  Godet  felt  that  cross-impact  analysis 
could  be  used  to  construct  and  rank  event  sequence  scenarios. 
In  recognition  of  the  fact  that  the  estimators  could  be 
inconsistent  in  stating  probabilities,  they  attempted  to 
derive  "theoretical"  scenario  probabilities  which  were 
implied  by  the  estimator  responses.  Using  those  implied 
probabilities  they  attempted  to  revise  the  estimated 
responses  to  fit  the  theoretical  scenario  probabilities 
and  produce  a consistent  set  of  estimate.  The  following 
diagram  summarizes  their  procedure  [Ref.  48,  p.  306]. 


Figure  2 

THE  SMIC-74  PROCESS 
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In  order  to  accomplish  this  process  Duperrin  and 
Godet  developed  a minimization  program  to  minimize  the 
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difference  between  the  estimated  factors  P Ci | j ) resulting 
from  the  experts'  responses  and  the  theoretical  factors, 
P*(ilj)  and  P* (i) , expressed  in  terms  of  even  sequence  scenario 
probabilities,  , [Ref . 48,  p.  306].  The  rationale  behind 
this  approach  was  covered  in  the  Mitchell  and  Tydeman  re- 
view. The  Duperrin  and  Godet  procedure  utilized  a minimi- 
zation program  of  quadratic  form  with  linear  constraints 
[Ref.  48,  p.  306-7] . 

n ^2 

minimize  Z = F [P  (i  | j ) P ( j ) “ T t(ijk)7r.] 
ij  k=l  ^ 


n ^ r 2 

+ ^[P{i|3’)P(j)  - I sCijk)iT,] 

ij  k=l  ^ 


r 

subject  to  y = If  Tfv  ^ ^ 

k=l  ^ 


Obviously,  solving  this  program  would  involve  utilizing 
a large  computer  package. 

Furthermore,  according  to  Mitchell  and  Tydeman  the 
ranking  produced  is  not  unique  "in  that  the  quadratic 
programme  has  other  possible  solutions",  and  the  "number  of 
probaJaility  sets  which  satisfy  the  quadratic  is  potentially 
very  large"  [Ref.  49,  p.  64],  These  conclusions  led  Mitchell 
and  Tydeman  to  propose  their  approach  discussed  previously. 
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. Sequential  Approach  (Sarin) 


Sarin's  approach  is  concerned  primarily  with  event 
sequence  scenario  generation.  Sarin  derived  a larget  set 
of  classical  probability  constraints  which  information 
elicited  on  initial  and  conditional  probabilities  must 
satisfy  in  order  to  be  consistent  (e.g.,  for  a set  of  six 
events  there  are  sixty-four  conditions  the  initial  proba- 
bilities alone  must  satisfy  to  be  consistent.)  [Ref.  38, 
p.  54-5] . He  then  develops  a set  of  linear  equations  with 
which  to  compute  the  2*^  possible  event  sequence  outcome 
probabilities  and  satisfy  the  constraints  (Ref.  38,  p.  59-60]. 

Sarin  proposes  that  an  interactive  computer  program 
be  used  in  conducting  the  analysis.  The  program  checks  the 
estimator  inputs  for  consistency,  computes  and  supplies 
bounds  for  further  estimates,  and  uses  the  consistent  inputs 
to  solve  the  linear  equations  producing  the  scenario  proba- 
bilities (Ref.  38,  p.  58].  This  approach  solves  the  consis- 
tency problem  through  providing  respondents  with  immediate 
feedback. 

In  this  approach,  it  is  possible  to  consider  higher- 
order  interactions  with  the  computer's  aid.  Because  these 
interactions  are  often  so  tightly  bound  it  becomes  unnecessary 
to  estimate  them  [Ref.  38,  p.  59].  Sarin  also  concludes  that 
each  discrete  event  influences  only  two  scenarios,  or  event 
outcome  sequences,  and  it  is  therefore  possible  to  conduct 
sensitivity  analysis  through  considering,  and  changing 
initial  values  for,  only  the  events  which  influence  the  top 
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ranked  scenarios.  In  this  manner  it  is  possible  to  estab- 
lish bounds  on  P(i)  under  which  the  scenario  ranking  does 

not  change  [Ref.  38,  p.  60].  Similar  bounds  can  be  computed 

\ 

for  the  conditional  probabilities,  each  of  which  influence 
only  four  scenarios. 

Once  again,  the  need  for  an  interactive  computer 
program  makes  demonstrating  this  approach  infeasible  in  the 
current  study.  Furthermore,  no  mention  is  made  of  conducting 
the  other  types  of  analysis  (event  dominance,  sensitivity, 
and  tracing  causal  effects)  which  would  be  necessary  in  a 
problem  considering  short  term  effects.  Although  such 
analyses  could  undoubtedly  be  accomplished,  it  would  require 
making  major  modifications  to  the  program. 

3.  KISM  (Kane) 

In  1972,  Kane  advanced  a cross-impact  procedure 
which  dealt  with  developing  trends  rather  than  discrete 
events.  His  trend  variables  behaved  in  a logistic  fashion 
(as  Gordon  proposed) . In  this  formulation  the  net  effects 
of  the  cross-impact  coefficients  went  to  zero  as  the  trends 
reached  their  maximum  or  minimum  values  [Ref.  47,  p.  132]. 

If  these  values  are  restricted  to  a ramge  between  one  and 
zero,  the  impact  of  one  trend  on  another  is  given  by 
[Ref.  47,  p.  133] : 

x^{t  + At}  * (t)^i 


where 


1 * X - “ij'  ■'J 

p^(t)  - 

^ ^ I ■*■  “ij^  ^j 

or 

1 + At  I sum  of  negative  impacts  on  1 

^i^^^  " 1 + At  I sum  of  positive  impacts  on  | 

where 

t ■ time 

■ the  cross-impact  value 
x^  » trend  i 

Xj  ■ trend  j 

The  relationship  was  modelled  on  a computer  using  the 
differential  [Ref*  49,  p.  1331. 

dx.  n 

= - y a..x.x.  Inx. 

^ 13  1 D 1 

These  developments  can  be  modelled  for  any  time  increment 
and  analyzed  by  varying  the  initial  assumptions  regarding  the 
probabilities  of  the  trends  and  the  cross-impacts.  Initial 
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trend  and  cross-impact  values  were  obtained  as  in  the 

25 

standard  approaches  [Ref.  47,  p.  131]. 


25 

This  approach's  intuitive  appeal  is  apparent  from  the 
computer  output  generated,  which  shows  the  interactions  as 
they  develop  over  time.  A sample  output  for  a transportation 
study  is  presented  below  along  with  the  cross-impact  matrix 
from  which  it  was  derived.  The  values  in  the  matrix  are  a 
series  of  +'s  and  -'s  which  were  converted  directly  to  integer 
values.  Another  interesting  point  is  that  a trend  can  impact 
itself  in  this  approach.  Initial  probabilities  for  the 
trends  as  established  by  the  developer  were  [Ref.  47,  p.  137] : 

Auto  Use  * . 8 
Comfort  ■ . 6 
^nd  Convenience 
Freedom  - . 6 
Speed  ■ . 5 
Auto  Cost*  .2 


T.ihlf*  I 

ihe  I'nftiYii  *>f  PtiMIr  I n .•«, 


Lipinski  and  Tydeman  have  attempted  recently  to 
extend  KISM  to  include  a consideration  of  discrete  events, 
non-constant  trends,  and  their  interactions  [Ref.  50].  As 
with  the  other  approaches  discussed  in  this  section  this 
method  suffers  from  the  requirement  that  a large  computer 
package  be  available  for  demonstration. 


G.  MAJOR  APPROACH  EVALUATION 

Four  primary  approaches  to  cross-i.mpact  analysis  have 
been  identified  and  explained. 

The  Basic  Method  — Attempted  to  elicit  quantitative 

values  for  interactions  among  events  and  revised  the 
initial  probability  estimates  through  a quadratic 
manipulation.  The  cross-impact  concept  used  was 
incompletely  defined,  and  subject  to  several  inter- 
pretations . 

Enzer's  Modification  — Obtained  estimates  in  terms  of 

likelihood  ratios,  included  non-occurrence,  addressed 
pairwise  consistency,  and  revised  the  condit.  •>'' 
probability  estimates. 

The  Turoff  Approach  — Assumed  estimator  consistency, 
used  inputs  to  derive  correlation  coefficients, 
and  attempted  to  identify  a unique  scenario. 

Cross-Event  Analysis  — Derived  conditional  probability 
relationships  from  causal  time-dependent  estimates 
and  used  a linear  program  to  identify  and  rank 
scenarios . 

Although  the  major  problems  associated  with  each  of  these 
approaches  were  identified  in  the  discussion,  selecting  an 
approach  for  application  cannot  be  made  unless  the  approaches 
are  systematically  compared.  The  evaluation  criteria  developed 
in  Chapter  Two  will  be  used  for  this  purpose.  Those  criteria 
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1.  Is  the  event  scope  adequate? 

2.  Can  events  be  rated  on  iraportance/desirability? 

3.  What  types  of  analysis  are  permitted? 

4.  Is  the  approach  computer  adaptable? 

5.  Are  required  estimates  clearly  formulated? 

6.  Are  estimates  consistent,  or  can  consistency 
be  obtained? 

7.  What  manipulation  algorithm  is  used? 

8.  Is  the  procedure  logical/ justifiable? 

9.  How  is  consistency  obtained? 

10.  How  are  scenarios  generated,  can  they  be  ranked? 

11.  How  many  estimates  are  required? 

12.  What  types  of  computations  must  be  performed? 

13.  How  many  terms  are  involved? 

14 . How  many  calculations  are  required? 

1.  Evaluation 

The  following  table  summarizes  the  findings  discussed 
in  the  preceding  sections  in  relation  to  these  evaluation 
criteria.  The  four  primary  cross-impact  approaches  identi- 
fied form  the  column  headings.  Each  entry  in  the  table  repre- 
sents a brief  summary  of  how  an  approach  was  evaluated  as  to 
meeting  the  numbered  criteria  which  form  the  row  headings. 
Specific  conclusions  based  on  this  evaluation  will  be  pre- 
sented in  the  selection  section  of  this  chapter  which  follows 
Table  One. 

A few  of  the  criteria  were  omitted  from  the  table 
because  they  were  met  equally  well  by  all  approaches.  These 
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APPROACH  EVALUATION 
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general  criteria  will  be  conunented  on  below.  In  considering 
the  last  three  criteria  which  addressed  the  required  compu- 
tations' form  and  number  (criteria  12,  13,  and  14),  two 
assumptions  were  made;  (1)  The  types  of  computations  and 
number  of  terms  (criteria  12  and  13)  include  all  possible 
computations  which  could  be  necessary  (including  computing 
cross-impacts  and  resolving  inconsistencies)  in  demonstrating 
the  approach,  except  for  those  involved  in  aggregating  esti- 
mates to  produce  an  average  value.  These  computations 
would  not  be  approach  dependent.  (2)  The  number  of  computa- 
tions (criterion  14)  does  not  include  computations  required 
in  either  aggregation  nor  resolving  inconsistencies,  as  the 
latter  depends  on  the  knowledge  of  the  respondents. 

The  general  criteria  were  found  to  be : 

a.  Event  Scope  (1) 

Event  formulation  in  all  the  methods  can  cover 
any  subject  or  combination  of  subjects.  The  only  restric- 
tions concern  the  linkages  between  events,  and  the  way  in 
which  events  are  stated.  These  restrictions  apply  equally 
to  all  approaches. 

b.  Event  Rating  (2) 

Although  each  approach  did  not  rate  events  for 
selection  for  analysis,  the  procedures  proposed  by  Enzer  and 
by  Mitchell  and  Tydeman  are  flexible  enough  to  be  applied  in 
any  cross-impact  analysis.  Thus  once  a technique  is  selected 
it  is  necessary  only  to  select  a method  to  evalute  the  events. 
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and  explain  the  criteria  for  subjective  estimates  to  the 
respondents . 

c.  Computer  Adaptability  (4) 

Each  method  reviewed  acknowledged  that  a computer 
would  be  necessary  in  order  to  deal  with  a large  event  set. 
Turoff’s  approach  required  that  an  interactive  computer  be 
available  to  query  respondents,  and  present  them  with  the 
results  produced  by  their  estimates.  Enzer  proposed  that  a 
computer  be  used  to  provide  probability  bounds  based  on 
estimator  responses  for  each  event  pair.  It  appears  that 
all  necessary  calculations  could  be  easily  programmed,  and 
that  a software  package  could  be  developed  to  elicit  responses 
for  any  type  of  event.  Therefore,  it  is  felt  that  all  the 
approaches  may  be  considered  computer  adaptable. 

The  evaluation  of  each  approach  in  relation  to 
the  remaining  criteria  is  presented  in  Table  One.  This 
evaluation  highlights  the  major  differences  between  each 
approach . 

2.  Selection 

As  the  discussion  below  indicates,  of  the  four  major 
cross-impact  approaches  evaluated,  the  Turoff  method  seems 
to  be  the  least  promising  for  demonstration  in  this  study. 

The  basic  (Gordon)  method  was  developmental  and  substantially 
extended  by  Enzer.  Enzer ’s  approach,  however,  involved 
difficulties  in  relating  respondents'  estimtes  (which  could 
be  interpreted  as  providing  causal  time-dependent  probabilities) 
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with  classical  probability  theory.  This  shortcoming  would 
medce  it  difficult  to  explain  the  technique  to  respondents. 

In  contrast,  the  Mitchell  and  Tydeman  approach  develops 
procedures  to  calculate  classical  conditional  probabilities 
from  the  apparently  time-dependent  estimates,  and  provides 
a scenario  ranking  algorithm.  For  these  reasons  it  was 
selected  for  demonstration  in  this  study.  The  following 
discussion  will  present  the  rationale  behind  these  conclusions, 
and  indicate  anticipated  problems  in  implementing  the  selected 
approach . 

Due  to  its  assumption  that  estimators  are  consis- 
tent, the  Turoff  approach  requires  that  they  be  allowed  to 
revise  their  estimates  until  satisfied  with  the  model  pro- 
duced. This  requirement,  as  mentioned  previously,  would  be 
hard  to  meet  in  a manual  demonstration.  The  assumption  of 
consistency  cannot  be  validated  because  the  cross-impacts 
are  calculated  in  terms  of  correlation  coefficients  (ranging, 
however,  from  +■»  to  -<»)  , which  do  not  reveal  inherent  incon- 
sistencies as  do  probabilities  when  they  exceed  unity  or 
become  negative. 

The  Turoff  approach  is  further  limited  by  its  time- 
independence  assumption  for  scenario  generation.  This  pro- 
duces only  one  scenario  for  each  set  of  estimates,  and  there 
is  no  reasonable  basis  for  assuming  that  it  is  either  a 
unique  or  most  probable  scenario.  Finally,  the  time- 
independence  assumption  would  be  unworkable  in  an  analysis 
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attempting  to  exaunine  the  effects  produced  by  discrete 
response  options  which  might  be  implemented  in  reacti.-g  to 
a developing  crisis  situation.  The  rationale  and  structure 
of  this  type  of  analysis  will  be  developed  more  fully  in  the 
extension  section  of  this  chapter  which  follows. 

The  basic  cross-impact  approach,  developed  by  Gordon 
and  Hayward,  while  important  in  introducing  the  technique's 
basic  concepts  and  potential  applications  was  primarily  a 
developmental  approach.  As  such  it  suffered  from  several 
conceptual  shortcomings  which  were  eliminated  in  later 
approaches.  These  shortcomings  were  reviewed  extensively 
elsewhere  in  this  study,  and  stem  from  the  use  of  the  quadratic 
manipulation  technique,  ignoring  non-occurrence,  and  incom- 
plete explanation  of  the  cross-impact  factors.  Because  this 
approach  was  greatly  modified  by  Enzer  it  was  decided  to 
eliminate  it  from  fxirther  consideration  for  selection. 

Enzer 's  modified  approach  dealt  with  many  of  the 
problems  inherent  in  the  basic  method.  He  developed  an 
uncertainty  measure,  addressed  non-occurrence  and  consistency, 
and  eliminated  the  quadratic  manipulation.  However,  in 
developing  these  improvements  he  failed  to  solve  the  problem 
of  how  to  interpret  estimator  responses.  Enzer  recognized 
that  causal  relationships  were  being  elicited,  but  used 
classical  probeibility  relationships  to  compute  non-occurrence 
and  resolve  inconsistencies.  The  cross-impacts  were  defined 
as  likelihood  ratios  denoting  an  event's  change  in  probability 
given  the  occurrence  of  another  event.  This  change  was 

108 


denoted  by  the  conditional  probability  relationship  P(i|j), 
however,  the  actual  relationship  seems  to  involve  time- 
dependence.  Therefore,  this  approach  seems  to  be  limited  by 
the  incomplete  incorporation  of  two  types  of  probabilistic 
relationships.  In  order  to  demonstrate  a technique  it  will 
be  necessary  to  explain  its  rationale  to  the  selected  study 
group.  This  requirement  makes  the  modified  approach  unsuit- 
able for  demonstration  because  of  the  difficulty  in  explain- 
ing these  fine  distinctions  and  their  implications  to  respon- 
dents unaccustomed  to  providing  this  type  of  information. 

Cross-event  analysis  assumes  that  respondents  normally 
think  in  terms  of  time-dependent  probabilities,  and  uses 
these  estimates  to  calculate  the  conditional  probabilities 
from  the  relationships  implied  by  the  estimator's  responses. 
Because  this  approach  seeks  to  explicitly  identify  quanti- 
tative values  for  all  possible  outcomes  of  each  event  pair, 
a greater  number  of  subjective  estimates  is  required.  How- 
ever, the  questions  used  to  obtain  these  estimates  are  formu- 
lated in  a manner  which  appears  to  be  more  closely  akin  to 
suspected  individual  thought  processes  ^ 1 j first ^ ^ ‘ 

Since  these  estimates  are  used  to  compute  the  proba- 
bilities of  the  eight  possible  ordered  outcomes  and  then  to 
calculate  the  classical  conditional  probabilities,  more  com- 
putational steps  are  involved  in  using  this  procedure.  These 
formulae  are  relatively  basic  and  can  be  easily  programmed 
on  a hand-held  calculator.  This  extra  burden  is,  therefore, 
not  considered  excessive  in  light  of  the  advantages  gained. 
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Using  the  time-dependent  causal  estimates,  pairwise 
probabilistic  consistency  is  automatically  checked  when  com- 


puting the  probabilities  of  the  ordered  outcomes.  By 
specifying  values  for  these  outcomes  it  is  possible  to  derive 
a logical  transition  to  the  classical  conditional  proba- 
bilities. This  avoids  the  problem  of  applying  Bayesian  proba- 
bility relationships  to  causal  estimates,  and  eliminates  the 
need  to  adopt  less  supportable  assumptions  - such  as  estimator 
consistency.  At  the  same  time,  because  it  is  possible  to 
use  the  classical  probability  relationships,  the  Mitchell- 
Tydeman  method  is  given  a firm  methodological  footing. 

Finally,  Mitchell  and  Tydeman ' s cross-event  approach 
provides  an  algorithm  for  computing  scenario  probabilities. 
This  capability  is  necessary  if  decision-makers  are  to  be 
provided  a means  of  assessing  possible  courses  of  action  in 
responding  to  ongoing  developments.  Scenario  probabilities 
cannot  be  obtained  under  the  Turoff  method’s  assumption  of 
time- independence , and  could  only  be  derived  from  the  Enzer 
method  by  conducting  an  excessive  number  of  Monte  Carlo  runs 
(exceeding  the  number  of  possible  event  sequences) . 

For  these  reasons,  the  Mitchell  and  Tydeman  cross- 
event analysis  method  was  selected  for  demonstration  and 
evaluation  in  this  study.  Because  the  purpose  of  the  demon- 
stration is  to  evaluate  cross-impact  analysis'  potential  as 
a decision  aid  in  crisis  situations,  several  modificatioi.:; 
to  the  approach  were  considered  necessary.  These  modifica- 
tions, presented  below,  were  intended  to  extend  the  method 
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in  order  to  evaluate  policy  options  for  response  in  a 
developing  situation. 

H.  EXTENSION 

This  section  will  discuss  several  methodological 
alterations  through  which  cross-event  analysis  can  be 
extended  as  a decision-aid  for  use  in  crisis  situations. 

Four  aspects  of  the  methodology  will  be  dealt  with.  These 
are:  (1)  the  time  span  of  analysis,  (2)  identifying  policy 

options  for  inclusion  in  the  event  set,  (3)  the  method  of 
analysis,  and  (4)  scenario  generation. 

1 . Time  Span 

Crisis  situations  normally  occur  without  adequate 
warning  and  allow  a short  time  for  response.  As  Enzer  and 
Alter  pointed  out,  the  time  span  usually  considered  in  a 
cross- impact  analysis  is  determined  by  the  amount  of  time 
required  for  an  event's  impact  to  be  felt  by  another  event 
(Ref.  40,  p.  2331.  Since  the  technique  being  explored  here 
is  intended  to  evaluate  response  options,  it  is  felt  that 
this  time  span  should  be  kept  relatively  short. 

In  a crisis  situation,  it  seems  reasonable  to  assume 
that  most  response  options  would  be  designed  to  have  an 
immediate  effect  in  altering  the  situation  in  some  desired 
direction.  For  this  reason,  it  appears  desirable  to  limit 
the  analysis  to  a consideration  of  the  two  weeks  or  so  follow- 
ing the  identification  of  a crisis.  As  there  is  almost  no 
time  lag  involved,  the  event  prediction  need  only  involve 
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occurrence  and  non-occurrence,  rather  than  exact  dates.  This 
is  a level  one  analysis  described  in  Johnson  [Ref.  40], 
which  predicts  only  those  events  which  occur  by  a specific 
date.  In  providing  initial  estimates,  respondents  will  not 
be  asked  for  occurrence  dates,  in  providing  causal  time- 
dependent  estimates  any  event  may  be  considered  to  occur 
before  any  other. 

In  the  analysis,  the  sequence  in  which  events  occur 
will  be  revealed  through  scenario  generation.  As  will  be 
shown  below,  this  will  aid  in  evaluating  likely  scenarios  for 
plausabili ty . This  assumption  would  not  be  valid  if  the 
effects  of  policy  options  were  to  be  examined  in  their  full 
context.  As  has  been  noted,  careful  evaluation  would  require 
an  assessment  of  a response  option's  long-term  effect,  not 
only  on  the  crisis  situation.  In  a "real  world"  application 
the  short-term  and  long-term  analyses  could  be  conducted 
simultaneously  to  identify  a possible  response  to  control  the 
crisis  and  to  evaluate  its  potential  future  effects.  Such 
an  approach  was  not  adopted  here  because  of  this  study's 
developmental  and  exploratory  character. 

2 . Identifying  Policy  Options 

The  normal  procedure  in  cross-impact  analysis  has 
been  to  evaluate  the  effects  produced  by  implementing  policy 
options  through  arbitrarily  increasing  or  decreasing  selected 
events'  initial  probabilities.  This  seems  to  be  contrary  to 
a basic  goal  of  the  approach,  which  is  to  identify  an  event's 
unanticipated  effects.  Furthennore,  Gordon  and  Hayward 


f 


! 

1 

hypothesized  that  an  event  normally  has  a predecessor  or 
precedes  another  event  [Ref.  26].  It  was  the  purpose  of  the 
technique  to  systematically  assess  these  interactions.  In 
the  normal  sensitivity  analysis,  a policy  option  is  excluded 
from  the  matrix,  and  assumed  to  have  only  the  effect  intended. 

In  the  Mitchell  and  Tydeman  approach,  subjective 
assessments  are  made  of  the  impact  that  specific  strategies 
have  on  the  various  scenarios.  This  requires  a prior  "Sys- 
tem Impact  Analysis"  to  assess  how  the  occurrence  of  a 
scenario  impacts  identifiable  system  attributes,  and  the 
development  of  "objective  criteria"  to  evaluate  policy 
strategies.  It  has  been  pointed  out  that  the  validity  of 
this  process  would  be  tenuous  for  problems  involving  national 
objectives.  Furthermore,  it  increases  the  number  and  com- 
plexity of  the  subjective  estimates  required. 

In  an  effort  to  siinpli*/  the  process  of  assessing  the 
impact  of  policy  options  cn  sicuational  outcomes,  this 
study  will  attempt  to  identify  specific  policy  responses 
available  for  dealing  with  the  hypothetical  situation  and 
include  them  in  the  original  cross-event  matrix.  These 
response  options  will  be  identified  and  assessed  as  any  other 
event  which  might  develop  from  the  given  situation  under 
normal  circumstances.  Using  this  approach  will  eliminate 
the  need  for  "System  Impact  Analysis".  Normal  sensitivity 
analysis  will  still  be  conducted  to  determine  response  option 
flexibility  in  the  event  that  available  information  was  mis- 
leading and/or  the  problem  components  were  evaluated  incorrectly 
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by  the  respondents.  Including  response  options  in  the  event 
set  also  permits  other  extensions. 

3.  Method  of  Analysis 

One  possible  extension  involves  the  type  of  analysis 
performed  by  evaluating  the  causal  relationships  contained 
in  the  cross-event  matrix.  If  the  events  representing  policy 
options  are  separated  from  the  events  representing  situation 
developments  a four-quadrant  matrix  is  formed  as  represented 
below. 


FIGURE  THREE 

FOUR-QUADRANT  CROSS-EVENT  MATRIX 


(note:  the  diagonal  cells  can  still  contain  the  initial 

probabi li tie  s ) 


Because  the  subjective  estimates  used  to  obtain  the 
conditional  probabilities  are  in  a causal  time-dependent 
form,  the  following  interpretations  might  be  given  to  each 
of  the  four  quadrants. 
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(dId)  — Represents  the  situation  as  it  might 
develop  assuming  no  corrective  action  is  taken. 
This  depicts  the  results  of  allowing  a given 
crisis  to  continue. 

(0 1 D)  — Represents  how  the  occurrence  of  given 
developments  effect  the  probabilities  of  various 
response  options . The  occurrence  of  certain 
developments  may  make  a response  option  more 
attractive.  (It  is  felt  that  at  this  point  not 
enough  information  is  available  to  use  this 
interaction  to  predict  which  response  option 
may  be  selected,  only  whether  or  not  the 
respondents  consider  it  more  attractive.) 

(d|0)  — Represents  the  effect  on  the  situation 
produced  by  the  occurrence  of  each  response 
option.  This  would  be  the  primary  quadrant 
with  which  to  evaluate  the  response  options. 

(0|0)  — Indicates  the  interaction  among 
possible  response  options.  This  possibly  could 
be  used  to  trace  the  impacts  of  a mixed 
strategy,  or  to  evaluate  the  effects  of  imple- 
menting more  than  one  response. 

As  in  the  traditional  approaches,  each  of  these 
quadrants  could  be  evaluated  to  determine  event  dominance, 
sensitivity,  and  estimator  uncertainty.  Other  forms  of 
analysis  would  remain  essentially  the  same . These  inter- 
pretations can  only  be  hypothesized  at  this  time.  It  will 
require  experimental  data  to  determine  their  applicability. 
Therefore,  this  type  of  analysis  will  be  attempted  using  the 
results  from  the  demonstration  exercise. 

4 . Scenario  Generation 

Since  specific  policy  options  are  to  be  included 
in  the  event  set,  it  is  questionable  as  to  whether  they 
should  be  included  in  the  event  sequence  scenario  generation. 
The  occurrence  or  non-occ\irrence  of  a policy  option  is  not 
determined  solely  by  the  situation  developments  but  also  by 
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the  decision-maker's  perceptions  and  the  evaluations  they 
make.  The  policy  options  are  explicitly  included  in  order 
to  facilitate  more  complete  evaluation.  Based  on  this 
information,  the  decision-makers  would  then  select  the 
option (s)  to  be  implemented  (i.e.,  decide  occurrence). 

It  might  also  be  argued  that  by  including  one,  or 
all,  options  in  event  sequence  scenario  generation  it  would 
be  possible  to  determine  which  option (s)  should  be  implemented 
(based  on  their  occurrence  or  non-occurrence  in  the  most 
probable  scenario).  In  other  words,  generating  and  ranking 
scenarios  with  N options  (N  = 1,2, 3, . . . ,n) , and  evaluating 
these  scenarios  for  desirability  would  indicate  which 
options  produced  the  most  desirable  outcomes. 

However,  in  a short  term  situation,  the  specific 
event  sequence  scenario  may  not  be  so  important.  Accurate 
information  is  limited,  and  the  unexpected  could  always 
happen  (including  events  not  specified  in  the  set)  . There- 
fore, scenario  generation  intended  to  provide  specific  predic- 
tions would  be  of  questionable  value  unless  the  problem  was 
clearly  understood  by  the  estimators.  In  view  of  these 
considerations,  this  demonstration  will  only  include  the 
situation  developments  in  event  sequence  scenario  generation. 

I.  CONCLUSIONS 

This  chapter  described  four  major  approaches  to  conducting 
cross-impact  analysis,  including  their  basic  assumptions, 
computational  procedures,  and  types  of  analysis  performed. 
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Based  on  this  discussion,  and  using  consistent  evaluation 
criteria,  the  approaches  were  compared  and  the  Mitchell- 
Tydeman  cross-event  analysis  was  selected  for  us  in  this 
study. 

Cross-event  analysis  was  selected  because,  although  it 
required  obtaining  more  numerous  subjective  estimate;?,  its 
methodology  and  underlying  assumptions  seemed  more  systematic 
and  thorough  as  well  as  intuitively  pleasing.  In  addition, 
the  calculations  required  in  the  procedure  appeared  to  be 
simple  enough  to  perform  without  using  a computer.  The 
Mitchell  and  Tydeman  approach  was  extended  for  application 
in  a crisis  situation.  This  extension  involved:  limiting  the 
time  span  considered  in  the  analysis,  including  policy  options 
as  discrete  events  for  analysis  in  the  matrix  rather  than 
through  sensitivity  analysis  alone,  and  restricting  scenario 
generation  to  include  only  situational  developments  while 
excluding  policy  options. 

As  indicated  in  the  discussion  of  the  major  approaches, 
several  problems  may  still  be  encountered  in  operationalizing 
this  approach.  Some  of  these  problems  are  inherent  in 
conceptualizing  the  cross-impact  technique  itself  and  can- 
not, therefore,  be  resolved  until  data  have  been  collected 
and  analyzed.  Such  problems  include:  how  estimators  can 
best  quantify  the  interrelationships  being  sought  (proba- 
bilistically or  as  correlations) , how  to  interpret  estimator 
responses  (as  classical  probabilities,  causal  time-dependent 
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probabilities,  or  as  correlation  coefficients) , and  how 
inconsistencies  should  be  treated  (ignored,  boundary  rules, 
or  fitting  methods) . The  selected  technique  assumes  respon- 
dents can  think  in  prob2djilistic  terms,  indeed,  in  terms  of 
causal  time- dependent  probabilities,  and  that  inconsistencies 
are  best  resolved  using  fitting  procedures. 

Other  potential  problems  arise  from  dealing  with  subjec- 
tive estimates,  and  from  the  methodology  used  in  the  selected 
approach.  These  include  how  best  to  aggregate  estimates, 
and  how  to  resolve  inconsistencies.  Both  of  these  depend  on 
the  form  in  which  the  estimates  are  elicited  and  the  actual 
values  obtained.  Therefore,  these  problems  will  be  examined 
in  the  context  of  the  selected  technique's  operationalization. 

The  next  chapter  will  discuss  the  actual  implementation 
process  and  present  the  demonstration  results.  The  imple- 
mentation process  will  describe  how  the  demonstration  was 
structured,  and  the  problems  encountered  in  using  the  Mitchell 
and  Tydeman  approach.  The  demonstration  results  will  be  used 
to  evalute  the  accvuracy  of  the  approach's  assumptions,  and 
its  potential  utility  as  a crisis  decision-aid. 
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VI.  OPERATIONALIZATION 


A.  INTRODUCTION 

Employing  cross-impact  analysis  in  an  experimental  appli- 
cation requires  developing  and  following  a logical  implemen- 
tation process.  Due  to  the  approach's  information  require- 
ments and  the  type  of  problem  proposed,  four  preliminary 
steps  are  necessary  for  implementation.  The  first  task  is 
to  identify  a study  group  to  act  as  expert  respondents.  Next, 
a problem  environment  for  analysis  has  to  be  constructed. 

Then,  this  problem  environment,  and  the  approach's  purpose, 
methodology,  and  information  requirements  have  to  be  explained 
to  the  study  group.  Finally,  a procedure  to  elicit  the  group's 
subjective  estimtes  has  to  be  devised. 

Once  the  preliminary  background  has  been  laid,  actual 
implementation  requires  collecting  and  structuring  the  res- 
spondents'  estimates  into  the  format  required  by  the  cross- 
event approach,  and  conducting  the  analysis.  This  process 
includes:  identifying  candidate  events  and  selecting  from 

these  the  analysis  event  set,  and  aggregating  individual  esti- 
mates into  group  response  values  to  provide  the  data  for 
replicating  the  analysis  performed  by  Mitchell  and  Tydeman 
(Ref.  29].^® 


26 

The  purpose  behind  using  group  values  is  two- fold. 
First,  it  should  reduce  instances  of  pairwise  inconsistency 
(Ref.  29].  Secondly,  as  proposed  by  Enzer,  the  respondents' 
opinion  spread  could  serve  as  an  estimator  uncertainty 
indicator  (Ref.  27,  p.  55]. 
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To  replicate  the  cross-event  analysis,  as  extended  in 
the  previous  chapter,  it  is  necessary  to  perforin  two  calcu- 
lation steps.  The  first  step  is  to  compute  the  event  pair 
ordered  outcome  probabilities  through  Pg)  . These  values 

are  then  used  in  the  second  step  which  calculates  the  con- 
ditional probabilities  implied  by  the  respondents'  estimates. 
The  conditional  probabilities  can  then  be  arranged  into  a 
cross-event  matrix  and  scenario  generation  undertaken.  These 
tools  are  then  used  to  perform  the  problem  analysis. 

This  chapter  describes  the  implementation  design  developed 
for  this  study.  It  discusses  how  the  actual  operationaliza- 
tion evolved  by  reviewing  the  successes  and  failures  encoun- 
tered in  following  the  designed  implementation.  Finally,  the 
data  collected  are  presented  and  analyzed. 

B.  IMPLEMENTATION  DESIGN 

The  entire  operationalization  process  yias  designed  to 
follow  the  demonstration  described  by  Mitchell  and  Tydeman 
[Ref.  29,  p.  145-147].  Their  "event  assessment"  procedure 
was  used  to  select  an  analysis  event  set  [Ref.  23,  p.  6-9], 
the  forms  for  obtaining  subjective  estimates  were  taken  from 
the  examples  contained  in  Ref.  29,  and  the  equations  used 
in  all  calculations  were  from  the  derivations  presented  in 
the  same  source. 

The  procedure  used  to  demonstrate  cross-event  analysis  in 
a hypothetical  situation  was  constructed  to  follow  five  phases. 
Each  phase's  purpose  and  structure  will  be  discussed  below. 


1.  Scenario  Construction 


To  establish  the  problem  environment  for  analysis  a 
hypothetical  problem  scenario  was  developed.  Several  con- 
siderations influenced  this  development.  Because  the  problem 
was  intended  to  represent  an  emerging  crisis  situation,  the 
scenario  had  to  threaten  identifiable  national  interests, 
suggest  potential  policy  options  which  could  be  identified 
as  responses,  and  involve  recognized  actors.  This  last  con- 
sideration was  adopted  for  two  reasons.  Using  real  actors 
would  provide  an  actual  basis  for  the  respondents'  percep- 
tions rather  than  having  to  structure  perceptions  by  speci- 
fying a host  of  hypothetical  historical,  social,  political, 
economic,  etc.  data  in  the  scenario.  Also,  using  real 
actors  was  expected  to  make  estimation  and  analysis  more 
meaningful  to  the  respondents. 

To  satisfy  the  above  requirements  and  involve  as  much 
current  information  as  possible  in  the  problem  environment, 
the  scenario  was  based  on  developments  in  South  Asia.  A 
brief  justification  for  this  decision  will  be  presented  in 
the  next  section  which  describes  the  actual  scenario  pre- 
sented to  the  respondents . 

The  scenario  was  presented  in  four  sections  to  allow 
as  complete  a development  as  possible.  Those  sections  were: 

(1)  Background  — described  the  primary  actors,  their  internal 
situations  and  external  relationships.  (2)  Event  Sequence  — 
listed  the  hypothetical  moves  and  counter-moves  taken  by  the 
various  actors.  (3)  Resulting  Situation  — set  forth  the 
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actual  problem  (crisis)  to  be  analyzed  and  responded  to. 

(4)  U.S.  Concerns  — described  what  appeared  to  be  the 
significant  U.S.  interests  in  the  area.  The  scenario 
presentation  was  conducted  as  the  second  segment  of  the 
design  implementation's  explanation  phase. 

2 . Explanation 

This  phase  described  the  experiment's  purpose  and 
methodology  to  the  study  group.  This  explanation  was  in- 
tended to  help  the  respondents  understand  their  role  in  the 
experiment.  It  included:  The  rationale  behind  using  sub- 
jective estimates,  the  information  the  respondents  would 
be  asked  to  provide,  the  information  format,  how  the  infor- 
mation would  be  analyzed,  and  answered  any  questions  which 
arose.  In  this  manner  it  was  hoped  to  fully  involve  the 
respondents  in  the  project  and  thereby  produce  more  care- 
fully considered  responses,  while  demonstrating  to  the 
respondents  the  approach's  potential  as  a learning  tool. 

The  explanation  phase  involved  two  segments.  The 
first  described  cross-impact  theory,  its  assumptions, 
methodology  and  proposed  applications.  Once  the  basic 
concepts  and  their  use  was  understood,  the  Mitchell  and 
Tydeman  approach  was  described  to  the  study  group.  This 
presentation  highlighted  the  functions  the  respondents 
would  be  required  to  perform. 

The  second  explcination  segment  presented  and  dis- 
cussed the  hypothetical  scenario.  This  was  presented  in 
four  sections,  as  previously  described,  in  order  that  the 
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participants  could  fully  understand  the  problem  elements 
involved.  In  an  actual  application  this  segment  would  be 
optional,  as  the  "experts"  could  be  expected  to  be  familiar 
with  the  problem  environment.  Once  the  respondents  under- 
stood the  technique  and  hypothetical  problem  the  third 
implementation  phase  was  begun. 

3 . Elicit  Estimates 

This  phase  was  designed  to  obtain  the  subjective 
estimates  used  in  the  problem  analysis.  Two  estimation 
procedures  were  developed  for  this  process.  The  first 
used  a "brainstorming"  technique  to  elicit  a candidate 
event  set.  Immediately  after  the  scenario  discussion  the 
respondents  were  asked  to  suggest  developments  which  might 
occur  resulting  from  the  proposed  situation.  All  subse- 
quent estimates  were  obtained  using  an  individual  response 
procedure . 

Once  the  candidate  event  set  was  established  the 
respondents  were  asked  to  individually  rate  each  event's 
initial  occurrence  probability  and  significance.  This 
information  was  used  to  conduct  an  "event  assessment"  and 
select  a manageable  event  set  for  analysis.  As  discussed 
previously,  this  procedure  was  adopted  to  reduce  both  the 
number  of  estimates  and  computations  required  by  concen- 
trating on  only  the  "important"  events.  Individual  esti- 
mates were  then  obtained  for  the  causal  time-dependent 
probability  relationships  through  C^)  existing  in  each 

event  pair.  These  data  were  intended  for  use  in  the  fourth 
implementation  phase. 
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4 . Conduct  Analysis 


The  fourth  phase  was  designed  to  structure  the 
respondents'  estimates  into  the  cross-event  format  and 
analyze  the  resulting  output.  Individual  responses  were 
aggregated  into  group  values  and  used  to  compute  the 
"cross-impacts"  from  the  formulae  developed  by  Mitchell 
and  Tydeman.  This  procedure  computed  the  ordered  event 
pair  outcome  probabilities  (P^  through  Pg)  and  the  resulting 
conditional  probabilities  for  each  event  pair.  At  the  same 
time,  pair-wise  consistency  could  be  checked  and  resolved 
if  necessary.  Because  group  values  were  used,  instances 
of  inconsistency  were  expected  to  be  rare. 

Given  consistent  estimates  a cross-event  matrix 
would  be  constructed  and  scenarios  generated.  These  data 
would  then  be  analyzed  as  described  in  Chpater  Three.  The 
analyses  to  be  performed  included;  For  events  — determining 
dominance,  sensitivity,  signficance,  and  incertainty . For 
interrelationships  — conducting  causal  tracings  through  the 
event  matrix  (as  discussed  in  the  extension  section  of  Chap- 
ter Three) , evaluating  response  options  using  the  matrix 
and  scenarios,  performing  scenario  and  sensitivity  analysis. 
The  implementation  design  included  presentation  of  the 
analysis  results  as  its  final  section. 

5.  Presentation  of  Results 

This  phase  was  designed  to  organize  and  present  the 
analysis  results  for  evaluation.  These  results  were  pre- 
sented to  the  study  group  to  close  the  feedback  loop  and 
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complement  the  learning  experience.  This  phase  also 
involved  the  participants  in  evaluating  the  cross-impact 
technique. 

Respondents  were  asked  to  suggest  possible  reasons 
for  the  results  obtained,  and  to  assess  whether  or  not 
the  information  derived  from  their  estimates  met  their 
expectations.  They  were  also  asked  to  comment  on  their 
impressions  concerning  the  technique's  procedures  and  poten- 
tial utility.  This  information,  and  the  analysis  results, 
were  used  in  evaluating  cross-event  analysis  as  a possible 
crisis  decision-aiding  methodology.  The  respondents' 
feedback  and  the  actual  implementation  experience  were  used 
to  evaluate  the  procedure's  ease  of  application,  to  iden- 
tify possible  problems  and  to  recommend  improvements.  The 
following  section  recounts  how  the  designed  implementation 
actually  evolved,  and  identifies  the  successes  and  problems 
encountered  in  operationalizing  cross-event  analysis. 

C.  ACTUAL  IMPLEMENTATION 

The  preliminary  implementation  steps  were  relatively 
easy  to  accomplish.  The  study  group  utilized  was  a Data 
Analysis  for  Naval  Intelligence  class  being  conducted 
(Spring  Quarter,  1979)  at  the  Naval  Postgraduate  School, 
Monterey,  California.  The  class  provided  ten  respondents 
(nine  Naval  officers  and  a civilian  Professor) . The  offi- 
cers had  various  types  of  operational  experience  in  the 
U.S.  Navy,  and  were  undertaking  graduate  studies  in  the 
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school's  National  Security  Affairs  Department.  Thus, 
their  background  was  felt  to  be  adequate  for  demonstration 
purposes.  All  respondents  had  some  exposure  to  probability 
and  statistics,  organizational  management  and  decision-making, 
and  various  topics  in  National  Security  and  Intelligence. 

In  addition,  most  had  kept  abreast  of  ongoing  developments 
in  South  Asia.  These  characteristics  simplified  the  explana- 
tion process  as  no  really  "new"  material  had  to  be  covered. 

The  explanation  phase  previously  outlined  consisted 
essentially  of  orally  summarizing  Chapter  Two  and  the  cross- 
event analysis  section  in  Chapter  Three  of  this  study.  The 
explanation  also  reviewed  the  procedure  designed  to  elicit 
the  group's  subjective  opinions.  In  the  second  explanation 
phase,  the  study  group  was  led  through  the  scenario  developed 
for  this  demonstration.  The  problem  scenario  presented  to 
the  respondents  will  be  described  below,  followed  by  des- 
criptions of  the  remaining  implementation  phases. 

1.  Scenario  Development 

Developments  in  South  Asia  and  around  the  Persian 
Gulf  have  become  a focal  point  for  U.S.  diplomatic,  economic, 
and  military  interests.  This  increasing  interest  has  been 
well  documented  by  noted  political  scientists  and  historians. 
These  authors  have  detailed  increasing  U.S.  interest  in. 
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and  cornmitment  to,  the  region  resulting  from  numerous 

27 

interrelated  factors. 

Some  of  the  primary  factors  influencing  U.S. 

interests  in  the  area  include,  increasing  U.S.  dependence 

on  imported  oil  and  the  tenuous  supply  links  through  which 
28 

it  is  obtained.  Another  factor  is  the  increased  Soviet 

influence  in  South  Asia,  their  increasing  naval  presence  in 

29 

the  Indian  Ocean,  and  the  purpose  behind  this  presence. 

A third  concern  is  the  continuing  tension  in  the  region 
which  seems  to  keep  the  relationships  in  the  area  in  a 


27 

For  example,  Nadav  Safran  discusses  the  U.S.  involve- 
ment growing  out  of  the  Middle  East  peace  process  [Ref.  51] . 
Richard  Ullman  has  examined  U.S.  policy  options  in  the 
area  in  light  of  continuing  tensions  and  outside  pressures 
[Ref.  523.  Basic  considerations  for  U.S.  policy  in  this 
area  were  presented  by  Campbell  and  Lenczowski  [Refs.  53 
and  54] . Inter-regional  relationships  and  super-power  roles 
are  excunined  by  Vali  [Ref.  55] , while  Wall  has  collected 
papers  addressing  relevant  strategic  concerns  [Ref.  56] . 

Many  of  the  interests  and  concerns  developed  by  these 
authors  are  reflected  in  official  U.S.  Government  docu- 
ments such  as  the  "Department  of  Defense  Authorization 
for  Appropriations  for  FY  79"  policy  statements  [Ref.  57]. 

28 

The  oil  question  is  explored  by  analysts  such  as 
Lenczowski  and  Anthony  [Refs.  58  and  59] . Its  future 
implications  are  examined  in  a CIA  forecast  [Ref.  60] , 
which  indicates  that  the  situation  is  unlikely  to  improve. 

29 

Increased  Soviet  influence  and  military  presence  is 
discussed  by  Lenczowski  [Ref.  61],  Hurewitz  [Ref.  62J , 
Tahtinen  [Ref.  633  r and  McConnell  [Ref.  64] . The  dispute 
over  the  intentions  behind  this  presence  has  produced  a 
resultant  U.S.  policy  debate  over  its  military  presence 
reviewed  in  Daniel  [Ref.  65]. 
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state  of  flux  and  instability.^^  The  complexities  of  these 
problems  are  indicated  by  several  authors.  Although  a complete 
development  of  these  and  associated  factors  is  beyond  the 
scope  of  this  study,  that  the  region's  importance  is  an  area 
for  continued  policy  analysis  should  be  self-evident. 

Further,  recent  developments  in  several  South  Asian  states  have 
thrust  the  region's  problems  more  prominantly  into  public  view. 

Among  those  recent  developments  is  the  pro-Soviet  coup 
in  Afghanistan  coupled  with  signs  of  increased  Soviet  influence 
within  the  new  regime  and  subsequent  internal  turmoil.  Another 
situation  causing  increased  concern  was  the  overthrow  of  the 
Shah  of  Iran  and  resulting  uncertainty  over  the  direction  in 
which  the  new  government  would  evolve.  Adding  to  general  con- 
cern for  the  course  of  the  Iranian  situation  has  been  the 
centrifugal  forces  set  in  motion  by  Kurdish  and  Arab  national- 
ists as  well  as  politically  liberal  forces  within  the  country. 
In  addition,  the  execution  of  Bhutto  by  Pakistani  authorities 
has  increased  internal  dissent  in  that  country.  This  dissent 
is  significant  because  it  occurs  during  a period  of  tension 
between  Afghanistan  and  Pakistan  concerning  reputed  Pakistani 
support  for  dissident  tribesmen  challenging  the  Taraki  regime 
in  Afghanis  tan . 


A regional  perspective  on  area  tensions  is  presented  by 
Amire  [Ref.  66],  and  Misra  [Ref.  67].  Burrell  and  Cottrell 
[Ref.  68]  explore  specific  problems  faced  by  the  states  of 
South  Asia. 

^^These  developments  are  being  continuously  reported  in 
magazine  and  newspaper  articles.  Articles  describing  these 
specific  developments  include:  On  the  Soviet  involvement  and 
turmoil  in  Afghanistan;  Sherwell,  Chris,  "Fighting  May  Boost 
Soviet  Presence  in  Afghanistan,"  The  Christian  Science  Monitor, 
p.  6,  Dec.  29,  1978,  and  Sherwell,  Chris,  ‘'Soviet  Union  Mired 
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For  these  reasons  it  was  decided  to  develop  a problem 
scenario  involving  relations  in  South  Asia.  The  study  group 
would  be  fcuniliar  with  the  basic  situation  existing  in  the  area, 
and  could  easily  identify  important  (as  well  as  plausible) 
developments  reflecting  U.S.  interests  and  possible  response 
options.  The  actual  scenario  was  developed  as  follows, 
a.  Background 

This  section,  supported  with  newspaper  clippings, 
presented  the  current  situation  existing  within  and  among  the 
three  regional  actors  being  considered.  Afghanistan  was 
described  as  having  recently  undergone  a pro-Soviet  coup.  Its 
military  was  identified  as  a prime  factor  in  maintaining  poli- 
tical control.  The  country’s  administrative  bodies,  including 
the  military,  were  reported  to  be  heavily  influenced  by  Soviet 
advisors.  In  addition,  the  state's  internal  situation  was 
characterized  as  being  plagued  with  anti-government  groups, 
consisting  mainly  of  Moslem  tribesmen  in  the  rural  areas. 

The  situation  in  Pakistan  was  described  as 
involving  increased  internal  dissent  resulting  from  Bhutto's 
execution.  Problems  were  also  reported  to  be  developing 
from  the  influx  of  Afghan  refugees.  In  addition,  the 
Pakistani  military  was  described  as  poorly  equipped  and 
preoccupied  with  the  Indian  threat  to  Kashmir. 

in  Afghanistan,"  The  Christian  Science  Monitor,  p.  3,  May  24,  1979. 
On  Iran's  internal  power  struggle;  Godell,  Geoffrey,  "Leftists 
Step  up  Struggle  for  Control  of  Iran  Revolution,"  The  Christian 
Science  Monitor,  p.  3,  Feb.  16,  1979,  and  Allaway,  Tony,  "Kurdish 
Unrest  Tests  New  Iran,"  The  Christian  Science  Monitor,  p.  3, 

Mar.  22,  1979.  On  Pakistan;  ''Delayed  Decision  on  Bhutto  to 
Affect  Pakistan  Deeply,"  The  Christian  Science  Monitor,  p.  4, 

Mar.  19,  1979,  and  Marshall,  Tyler,  *^Indian  Subcontinent  Faces 
Pressure  for  Arms  Buildup,"  The  Los  Angeles  Times,  p.  1, 

April  22,  1979. 
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Iran ' s current  government  was  described  as 
disorganized,  composed  of  several  rival  factions,  and 
challenged  by  elements  (including  leftists  and  ethnic 
minorities)  desiring  more  control.  The  effectiveness  and 
loyalty  of  Iran's  military  was  also  brought  into  question, 
due  to  its  history  of  supporting  the  Shah  and  the  execution 
of  several  important  leaders.  This  development  stressed  the 
fact  that  authority  in  Iran  could  still  be  considered  to 
be  in  a state  of  flux.  Finally,  Iran's  historic  role  in 
controlling  disputes  between  Afghanistan  and  Pakistan 
concerning  the  territory  of  Baluchistan  was  mentioned. 

Using  this  simplified  background  discussion  as  a basis, 
a hypothetical  sequence  of  developments  was  proposed  to 
the  study  group. 

b.  Event  Sequence 

- Tribesmen  in  Baluchistan  (South  Afghanistan) 
begin  a national  autonomy  movement.  This  movement 
receives  some  support  from  related  tribes  in  Iran  and 
Pakistan. 


- Clashes  occur  between  Afghani  government 
forces  and  rebel  tribesmen  in  Baluchistan. 

- The  Afghani  government  dispatches  additional 
troops  to  control  the  area.  These  troops  are  Soviet 
supported,  but  the  extent  of  this  support  is  unknown. 

- These  forces  drive  the  majority  of  the  rebels 
across  the  border  into  Iran  and  Pakistan. 

- Both  sides  (rebels  and  Afghani  troops)  begin 
cross-border  raids.  By  this  time  the  majority  of  the 
Afghan  army  is  committed. 

- Afghanistan  charges  that  Iran  and  Pakistan 
are  aiding  the  rebel  forces.  The  Afghan  government  vows  to 
defeat  any  outside  threat  to  its  territorial  integrity. 
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The  Soviet  Union  supports  this  delcaration  and  offers  to 
increase  assistance  to  Afghanistan. 

- Iran  authorities  deny  the  Afghani  charges  and 
state  that  increased  Soviet  involvement  would  only  worsen 
the  situation.  Pakistan  charges  the  Soviet  Union  with  trying 
to  precipitate  a crisis  in  the  region,  and  requests  U.S. 
aid. 


- Regular  Afghani  forces  clash  with  Iranian 
border  troops.  This  clash  erupts  into  uncontrolled  shelling 
along  the  border. 

- Baluchistan  rebels  call  on  Muslim  states  for 
support  against  the  communists,  suggesting  a "holy  war" 

be  launched. 


- Iranian  leftists,  reportedly  supplied  by  the 
Soviets,  begin  a terrorist  campaign  to  oust  the  Ayatollah's 
government . 


- Iran  announces  support  for  rebel  tribesmen 
and  launches  a "limited"  invasion  to  pacify  the  area,  and 
expel  "expansionist"  communists. 

- Pakistani  forces  join  the  conflict. 

- The  Soviet  Union  sends  troops  to  Afghanistan 
which  check  the  Iranians.  Reports  indicate  the  Soviets 
have  -mobilized  on  Iran's  border,  that  their  agents  are 
agitating  Iranian  Kurds,  and  that  they  have  sent  a secret 
mission  to  Iraq. 

- Soviet  and  Afghani  forces  press  attack  to  the 
original  Afghan  border,  but  show  no  signs  of  halting. 

Steps  are  being  taken  to  set  up  a "puppet"  government  for 
Baluchistan.  Indications  are  that  Soviets  and  Afghanis 
may  attempt  to  include  Baluchistan  land  in  Iran  and 
Pakistan  under  control  of  an  "autonomous"  Baluchi  government. 

Based  on  these  developments,  the  following 
situation  was  proposed  to  exist  in  South  Asia.  This  situa- 
tion described  the  crisis  problem  to  be  evaluated  using 
cross-event  analysis. 

c.  Resulting  Situation 

- Open  warfare  exists  in  the  area  south  of 
Afghanistan  extending  into  Iran  and  Pakistan.  This 
fighting  involves  Afghanistan,  Iran,  Pakistan,  rebel 
tribesmen,  and  (reportedly)  some  Soviet  forces. 


t 

! 

i! 

i i 

^ , 


131 


- The  Soviets  have  mobilized  along  Iran's 
border  and  are  conducting  secret  talks  with  Iraq. 

- Iranian  Kurds  and  leftists  (separately)  are 
almost  in  open  revolt. 

- India  has  mobilized  along  Pakistan's  border, 
to  "prevent  the  conflict  from  spilling  over." 

- Chinese  troops  have  mobilized  in  Sinkiang 
in  response  to  "Soviet  aggression,  and  to  demonstrate 
support  for  the  Baluchistan  People’s  Movement". 

To  guide  the  respondents  in  proposing  candidate 
events  the  following  "probable"  U.S.  concerns  were  also 
presented. 


d.  U.S.  Concerns 

- The  current  conflict  threatens  navigation  in 
the  Straits  of  Hormuz.  In  addition,  the  fighting  is  close 
to  the  main  oil  producing  region  in  Iran,  threatening  the 
security  of  those  fields. 

- The  Soviet  Union  may  extend  its  influence  to 
the  Indian  Ocean  if  it  manages  to  sponsor  a client  state  of 
Baluchistan.  This  would  permit  both  overland  access  to 
the  Indian  Ocean  and  possible  naval  basing  rights. 

- The  current  turmoil  directly  threatens  U.S. 
civilians  in  the  area  and,  to  a lesser  extent,  U.S.  economic 
interests. 


- The  potential  involvement  of  India  and  China 
and  possible  direct  Soviet  intervention  suggest  a major 
confrontation  may  be  developing. 

This  scenario  provided  the  background  against 
which  the  respondents  were  asked  to  suggest  a candidate 
event  set,  and  formulate  the  required  subjective  estimates. 
This  process  will  be  described  in  the  following  section 
which  details  the  implementation  design's  third  phase  as 
it  actually  developed. 
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2.  Elicit  Estimates 
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The  next  task  undertaken  in  the  demonstration  re- 
quired establishing  a candidate  event  set.  Before  the 
group  was  asked  to  brainstorm  possible  developments,  several 
procedural  points  were  reviewed.  The  study  group  was 
reminded  that  suggested  events  be  considered  as  developing 
from  the  given  situation.  Enzer's  criteria  for  including 
events  in  the  set  were  also  reviewed.  These  criteria 
stress  that  candidate  events  be  (1)  important  to  the  situa- 
tion, (2)  of  uncertain  outcome,  (3)  causally  related,  and 
(4)  specific.  Finally,  the  event  formulation  rules  proposed 
by  Mitchell  and  Tydeman  were  reviewed  to  assure  the  suggested 
events  could  be  clearly  understood  by  the  group.  These 
procedural  points  were  discussed  in  Chapter  Two. 
a.  Establishing  the  Event  Set 

Once  the  group  understood  the  required  event 
formulation  method,  a brainstorming  session  began.  As 
events  were  suggested  they  were  recorded  and  posted  before 
the  group.  This  process  continued  until  it  became  apparent 
that  the  participants  had  e.xhausted  all  their  ideas.  A 
brief  discussion  session  then  was  allowed  to  clear  up  any 
uncertainties.  Twenty-nine  candidate  events  were  proposed 
during  this  session.  Ten  of  these  represented  possible  U.S. 
responses  to  the  situation,  the  remainder  represented  situa- 
tion developments.  These  events  are  presented  in  .Appendix 
A. 
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In  order  to  select  an  analysis  event  set  the 
group  was  next  asked  to  provide  the  subjective  evaluations 
required  for  Mitchell  and  Tydeman's  "event  assessment" 
procedure.  These  inputs  consisted  of  two  estimates.  The 
first  required  the  respondents  to  evaluate  each  event's 
initial  occurrence  probability.  These  probabilities  were 
collected  individually  by  having  each  respondent  score  every 
event  on  the  following  scale.  Each  evaluation  was  to  be 
made  without  considering  the  influence  any  other  event  in 
the  set  might  have  on  the  situation. 


TABLE  TWO 

EVENT  PROBABILITY  DESCRIPTIONS  AND  RATINGS 


STATEMENT 

RATING 

Won ' t Occur 

.Q5 

Very  Unlikely 

.15 

Unlikely 

.30 

Fifty-Fifty 

.50 

Likely 

.70 

Very  Likely 

.85 

Certain 

.95 

The  second  estimate  required  respondents  to 
evaluate  each  event's  significance  should  it  occur.  An 
event's  significance  was  defined  as  the  influence  it  would 
have  on  the  overall  problem  situation,  assuming  it  did 
occur.  These  ratings  were  based  on  the  following  scale, 
and  were  again  collected  individually. 


TABLE  THREE 


EVENT  SIGNIFICANCE  DESCRIPTIONS  AND  RATINGS 


STATEMENT  RATING 


No  Importance  1 
Some  Importance  2 
Important  3 
Very  Important  4 
Critical  5 


This  procedure  produced  ten  values  under  each 
criterion  for  each  of  the  twenty-nine  events.  These  indi- 
vidual scores  were  aggregated  and  used  to  produce  an  impact 
rating  for  each  event.  The  twenty-nine  candidate  events 
and  their  probability,  significance,  and  impact  ratings  are 
contained  in  Appendix  A. 

Several  problems  were  encountered  during  this 
estimation  procedure  which  were  unanticipated  in  the  prob- 
lem design.  While  eliciting  candidate  events  two  tenden- 
cies were  noticed  which  seemed  to  inhibit  the  formulation 
process.  In  brainstorming,  the  respondents  seemed  to  con- 
centrate on  a suggested  event  and  propose  subsequent  events 
which  would  develop  should  the  first  event  occur.  Often 
this  process  would  lead  far  afield  until  interrupted  by 
the  facilitator  interjecting  an  alternative  event.  On 
other  occasions,  an  event's  suggestion  seemed  to  elicit  the 
proposal  of  a response  event  aimed  specifically  at  the  first 
event.  Another  problem  encountered  stemmed  from  the  respon- 
dent's experience  level  or  knowledge. 


Some  respondents  seemed  to  be  intimidated  by 
the  ease  with  which  other  respondents  formulated  and  jus- 
tified suggested  events.  A few  respondents  stated  they 
felt  unqualified  to  participate,  and  questioned  the  validity 
of  their  future  estimates  given  their  lack  of  equal  knowl- 
edge concerning  South  Asia.  These  problems  seem  to  be 
inherent  in  the  "brainstorming"  process. 

In  future  applications  it  is  felt  that  using 
trained  or  experienced  facilitators  could  reduce  the  prob- 
lem of  respondent  intimidation,  and  better  control  the  brain- 
storming process.  In  addition,  research  using  the  delphi 
procedure  indicates  that  having  respondents  rate  themselves 
on  their  confidence  in  answering  questions  provides  a cri- 
terion with  which  to  weight  individual  responses  to  produce 
improved  group  values  [Ref.  13] . This  method  might  be 
incorporated  in  future  cross-impact  applications. 

The  fact  that  suggested  events  often  led  to 
related  events  being  suggested  indicates  a basic  cross- 
event assumption  may  be  unfounded.  The  approach  operates 
on  the  theory  that  respondents  can  think  in  terms  of  dis- 
crete events.  Based  on  this  assumption  respondents  are 
asked  to  provide  initial  probability  and  significance  esti- 
mates for  each  event  independently.  However,  during  the 
brainstorming  process,  respondents  exhibited  a tendency  to 
propose  events  in  an  iterative  manner.  This  suggests  that 
the  group  may  find  it  difficult  to  conceptualize  relationships 
in  terms  of  discrete  events,  and  therefore  may  conceive  of 
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of  related  probability  and  significance  questions  inter- 
dependently,  or  as  contingencies.  This  in  turn  indicates 
that  another  approach  to  quantifying  estimates  may  be  more 
appropriate.  This  point  will  be  developed  more  exten- 
sively in  the  next  chapter. 

Other  problems  were  encountered  when  the  indi- 
vidual estimates  were  aggregates  to  produce  the  impact 
ratings.  In  their  procedure,  Mitchell  and  Tydeman  use  a 
weighted  product  of  each  event's  probability  and  signifi- 
cance scores  to  produce  its  impact  rating.  This  procedure 
was  found  to  produce  inadequate  results  in  this  demonstra- 
tion. The  results  obtained  seemed  to  disguise  the  impact 
assigned  an  event  which  received  a combination  of  a low 
and  high  rating  on  the  two  criteria  evaluated.  A simpli- 
fied, hypothetical  example  may  help  to  illustrate  this  point. 

Assume  three  countries  are  rated  on  their  mili- 
tary potential  by  aggregating  scores  for  their  Army,  Navy, 
and  Air  Force.  Using  a multiplicative  index  produces  the 
following  results. 

FIGURE  4 

results  of  a multiplicative  index 

MILITARY 

COUNTRY  ARMY  (x)  NAVY  (x)  AIR  FORCE  (=)  POTENTIAL 


A 

5 

2 

3 

30 

B 

5 

1 

3 

15 

C 

5 

0 

3 

0 
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Using  the  same  values  with  an  additive  index  seems  to 
produce  a more  realistic  measure. 


FIGURE  5 

RESULTS  OF  AN  ADDITIVE  INDEX 


COUNTRY 


A 


B 
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Another,  less  significant,  problem  developed 
because  two  different  scales  were  employed  in  the  "event 
assessment."  Probability  values  were  quantified  between 
zero  and  one,  while  significance  scores  ranged  from  one 
to  five.  This  further  distorted  the  multiplicative  index. 
As  a result  the  impact  scores  produced  an  analysis  event 
set  which  did  not  seem  to  accurately  reflect  the  event's 
relative  importance.  To  overcome  this  problem  two  correc- 
tions were  introduced  to  the  aggregation  procedure. 

The  significance  ratings  were  rescaled  to 
fall  between  zero  and  one.  The  final  ratings  employed 
were;  5=.95,  4^.7,  3=.5,  2=.3,  and  1=.05.  Then  the 
individual  probability  and  significance  ratings  were 
summed  to  produce  an  individual  event  impact  rating. 

All  three  ratings  were  then  averaged  to  produce  a group  or 
mean  score,  and  the  standard  deviations  were  computed  for 
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each  measure.  These  are  the  values  presented  in  Appendix 

A. 

This  correction  is  not  without  conceptual 
limitations.  The  assumption  implicit  in  summing  the  two 
evaluation  measures  is  that  both  probability  and  signifi- 
cance are  equally  important  in  determining  an  event's 
impact  on  the  problem  environment.  It  is  doubtful  whether 
this  assumption  would  hold  up  under  rigorous  examination. 

It  appears  that  an  event's  significance  would  be  a more 
important  consideration  than  its  occurrence  probability. 

This  is  because  an  event's  impact  has  no  meaning  should  it 
not  occur,  therefore,  it  only  becomes  important  from  a 
policy  standpoint  once  it  occurs,  and  this  importance 
derives  from  its  significance  to  the  situation.  This  prob- 
lem relates  to  Gordon's  discussion  of  the  complex  factors 
which  make  up  an  event's  cross-impact  [Ref.  24].  Presently 
there  exists  no  algorithm  relating  the  many  factors  (such 
as  probabilities  of  "feasibility"  and  "pursuit",  and  event 
significance)  which  compose  an  event's  impact.  As  a result 
simple  summing  was  the  only  approach  which  could  be  readily 
applied. 

Candidate  events  were  ranked  by  impact  scores 
and  six  events  selected  to  form  the  analysis  set.  A final 
ranking  by  event  impact  scores  is  contained  in  Appendix  A. 
Because  one  purpose  of  the  demonstration  was  to  examine  the 
cross-impact  technique's  applicability  to  evaluating  specific 
policy  response  options  these  six  events  were  not  chosen  from 
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the  top  ranked  candidate  events.  The  six  top  ranked  events 

included  only  one  event  identified  as  a response  option 

(number  three).  This  was  because,  while  most  response 

events  were  evaluated  as  having  high  initial  probabilities, 

32 

their  significance  was  evaluated  as  being  relatively  low. 

To  obtain  a matrix  containing  an  equal  number  of  response 
events  and  situation  development  events,  the  events  were 
separated  and  the  top  three  chosen  from  each  category. 

The  analysis  event  set  and  the  mean  initial 
probabilities  are  presented  below.  Six  events  were  used 
because  this  was  felt  to  be  the  maximum  number  which  could 
be  effectively  manipulated  by  hand.  It  should  be  noted  that 
the  initial  probability  values  have  been  rounded  to  two 
decimal  places.  The  averaging  technique  used  retained  three 
decimal  place  values  for  comparison  purposes.  Because 
these  values  were  averaged  subjective  estimates,  this 
degree  of  accuracy  would  not  be  jsutified  for  use  in  the 
actual  demonstration.  This  event  set  formed  the  basis  for 
the  remaining  individual  estimates,  that  is,  the  five 
causal  time-dependent  relationships  developed  by  Mitchell 
and  Tydeman. 


T2 

■ This  result  is  interesting  due  to  its  implications. 
Although  the  U.S.  is  assumed  to  have  an  extensive  number 
of  responses  available  which  could  be  employed  in  a given 
situation,  the  respondents  obviously  felt  that  the  majority 
of  these  would  be  ineffective  in  this  case.  Of  the  three 
response  options  included  in  the  analysis,  only  one  had  a 
significance  score  comparable  to  that  of  the  situation 
development  events. 


TABLE  FOUR 


SELECTED  EVENTS 


No. 

STATEMENT 

Avg. 

22 

Soviet/Afghan  forces  gain  control  of 
Baluchistan. 

.60 

28 

Soviet  Union  builds  up  Indian  Ocean 
squadron . 

.80 

23 

Soviet  Union  airlifts  supplies  to 
Afghanistan. 

.72 

3 

United  States  deploys  CTF  to 

Indian  Ocean. 

.76 

9 

United  States  evacuates  civilians 
from  area. 

CO 

• 

8 

United  States  offers  to  negotiate 
a cease-fire. 

.64 

b.  Estimate  Relationships 

The  six-event  set  produced  fifteen  possible 
event  pairs  which  required  further  estimation.  The 
required  estimates  were  the  five  time-dependent  probabili- 
ties through  C^)  which  assessed  occurrence,  non- 
occurrence, and  joint  occurrence.  Thus,  seventy-five 
estimates  were  required  from  each  respondent. 

In  order  to  systematically  produce  and  collect 
these  estimates,  response  forms  were  duplicated  from  the 
examples  provided  by  Mitchell  and  Tydeman  [Ref.  29,  p.  146- 
147].  The  forms  used  are  reproduced  on  the  next  page. 


FIGURE  6 

FORM  FOR  EXTIMATING  AND 

Would  the  prior  occurrence  of  event  # cause  your  estimate 
of  the  Probability  of  event  # to; 

Increase  Stay  the  same  Decrease 


If  your  estimate  of  event  # probability  is  changed  (increased 
or  decreased)  is  the  magnitude  of  the  change: 

Only  Slight  A Little  A Fair  Bit  A Lot  Sufficient  to 

make  event  # 
certain  or 
impossible 

5 10  25  50  100 

(%  of  total  possible  change) 

FIGURE  7 

FORM  FOR  ESTIMATING  AND 

2 4 


Would  the  prior  non-occurrence  of  event  # cause  your 
estimate  of  the  Probability  of  event  # to: 

Increase  Stay  the  Same  Decrease 


If  your  estimate  of  event  # probability  is  changed  Cincreased 
or  decreased)  is  the  magnitude  of  the  change: 

Only  Slight  A Little  A Fair  Bit  A Lot  Sufficient  to  make 

event  # certain 
or  impossible 

i 

5 10  25  50  100 

(%  of  total  possible  change) 
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FIGURE  8 


FORM  FOR  ESTIMATING 

Assuming  both  events  # and  # occur  within  the  next  14  days 

Event  is  certain  to  occur  first 
Event  is  very  likely  to  occur  first 
Event  is  likely  to  occur  first 

Event  and  event  are  equally  likely  to  occur  first 
Event  is  likely  to  occur  first 
Event  is  very  likely  to  occur  first  S 

Event is  certain  to  occur  first 

These  forms  were  organized  by  selecting  the 
first  event  in  the  set  and  pairing  it  with  the  remaining 
five  events.  The  next  series  of  estimates  paired  the  second 
event  with  the  remaining  four  events  and  so  on  until  all 
fifteen  event  pairs  had  been  specified.  Because  there  were 

j 

seventy-five  total  estimates,  the  forms  were  delivered  to 
the  respondents  in  two  batches.  Each  respondent  was  given 
two  days  to  complete  each  batch.  The  forms  were  then 
collected  and  sorted  by  event  pair  and  the  relationships 
assessed.  The  ten  responses  for  each  relationship  were 
averaged  to  produce  a mean  score  for  each  relationship. 

These  scores  are  contained  in  Appendix  A. 

Several  complaints  were  voiced  by  the  respon- 
dents concerning  this  estimation  procedure.  This  approach 
made  the  estimation  process  monotonous.  Three  questions 
were  repeated  with  only  different  event  numbers 
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being  inserted  to  complete  the  seventy-five  data  points. 

In  addition,  the  question  wording  was  felt  to  be  tortuous, 
specifically  through  use  of  the  terms  "prior  occurrence", 
and  "prior  non-occurrence". 

These  problems  result  from  manually  replicating 
the  cross-event  procedure.  It  was  not  possible  in  the 
time  available  to  devise  an  optimal  question  format  for 
each  event  pair.  Such  a format  could  have  incorporated 
the  event  statements  within  the  question.  For  example 
the  following  formulation  for  event  pair  (28,3)  would 
probably  have  been  more  acceptable: 

"Assuming  the  Soviet  Union  builds  up  its 
Indian  Ocean  fleet,  would  you  estimate 
the  probability  that  the  U.S.  deploys 
a CTF  to  the  Indian  Ocean  to:" 

and  for  non-occurrence: 

"Assuming  the  Soviet  Union  does  not 
build  up  its  Indian  Ocean  ?Teet,  would 
you  estimate  the  probability  ..." 

Such  a format  could  obviously  be  constructed  for  each 

event  pair.  Several  advantages  might  stem  from  adopting 

this  approach.  The  more  precise  wording  explicitly 

incorporates  the  event  statements  being  considered.  The 

causal  time-dependent  nature  of  the  relationship  being 

explored  is  implicit  in  the  question  without  employing 

phrases  such  as  "prior  occurrence."  As  a result  the  responses 

obtained  might  be  more  accurate  and  consistent,  however, 

this  fact  cannot  be  validated  without  further  analysis  of 

how  respondents  interpret  questions. 
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Another  problem  encountered  in  the  procedure 
was  not  voiced  by  the  respondents,  but  was  hypothesized 
to  account  for  the  responses  obtained  when  the  group  values 
were  analyzed.  The  analysis  of  the  responses  presented  in 
the  next  section  found  that  every  event  pair  was  inconsistent 
in  terms  of  the  through  Pg  ordered  outcome  probabilities 
computed.  This  problem  was  felt  to  result,  at  least  par- 
tially, from  the  sequence  in  which  estimates  had  been 
obtained. 

The  sequence  used  in  this  demonstration  was 
selected  because  it  systematically  specified  all  the  event 
pairs.  This  was  necessary  to  insure  that  all  respondents 
received  the  proper  forms  and  thus  provided  all  required 
estimates.  However,  this  may  have  been  an  inappropriate 
way  to  procede  through  the  event  pair  set  for  estimation 
purposes.  In  addition,  the  five  relationships  being  esti- 
mated were  not  presented  to  the  respondents  in  order.  The 
joint  occurrence  question  (C^)  being  retained  until  the 
end  of  each  series  (Event  1 j Event  2,...,  event  6)  of  the 
through  questions. 

As  the  respondents  answered  the  questions  in 
the  order  of  presentation,  it  is  possible  that  they  lost 
track  of  the  values  previously  specified.  It  is  also 
possible  that  the  respondents  were  prevented  from  con- 
sidering their  previous  responses  concerning  event  pairs 
which  might  have  been  felt  to  be  related  to  each  other. 

Once  again  these  possibilities  cannot  be  verified  without 
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experimental  analysis.  No  doubt,  there  is  an  optimal  way 
to  present  the  event  set  for  estimation. 

This  conclusion  is  based  on  two  arguments  pre- 
sented in  the  cross-impact  theory  discussion.  First,  Enzer 
suggested  that  probabilities  of  conditional  relationships 
were  bounded  by  the  probabilities  previously  specified  for 
that  event  pair.  If  this  is  true  then  this  condition  could 
also  apply  to  the  time-dependent  causal  probabilities.  The 
second  argument  is  even  more  basic  to  cross-impact  theory. 

Gordon  suggested  that  most  events  are  related 
in  some  way.  Enzer  approached  this  conclusion  in  his  dis- 
cussion of  coupling.  It  therefore  seems  reasonable  that  the 
discrete  events  composing  a set  may  all  be  coupled  in  some 
way.  This,  it  should  be  noted,  is  not  the  concept  of  higher 
order  interactions  which  Turoff  attempted  to  measure,  but 
refers  to  direct  relationships  aimong  all  events  in  the  set. 
Assuming  such  a situation  is  present,  theii  in  order  to  pro- 
duce a consistent  set  of  estimates,  a respondent  needs  to 
be  able  to  review  and  incorporate  not  only  his  previous 
estimates  within  an  event  pair  but  among  event  pairs  as  well. 

In  summary,  the  problems  encountered  and  sus- 
pected in  this  process  were:  CD  monotony,  (2)  imprecise 
question  wording,  and  t3)  inappropriate  question  sequence. 

To  overcome  these  problems  it  would  be  necessary  to  devise 
an  estimation  procedure  which:  (1)  formulates  questions  using 
the  event  statements,  (2)  develops  a more  exact  question 
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structure,  (3)  allows  estimators  to  review  their  responses 
as  they  estimate,  and  (4)  allows  estimators  to  compare 
their  responses  across  related  event  pairs. 

One  possible  approach  to  overcoming  these  prob- 
lems might  be  to  use  an  interactive  computer  program  to 
elicit  estimator  responses.  This  approach  was  demonstrated 
by  Turoff.  As  proposed  here,  the  program  would  allow  the 
respondent  to  make  estimates  on  the  basis  of  responses  con- 
cerning other  estimates.  Obvious  extensions  to  this  approach 
include  computing  probability  boundaries  as  suggested  by 
Dalkey,  and  displaying  results  to  allow  re-estimation  as 
suggested  by  Turoff.  Using  an  interactive  computer  would 
aid  in  obtaining  consistent  probability  estimates  (this 
problem  will  be  returned  to  in  a later  section) . It  might 
also  provide  useful  data  for  determining  how  respondents 
interpret  questions  posed,  and  for  devising  an  optimal 
estimation  sequence. 

Despite  the  problems  encountered  by  the  respon- 
dents, and  because  the  suspected  problems  were  not  immediately 
apparent,  the  respondents'  estimates  were  collected  and 
averaged  as  planned.  These  values  formed  the  input  for 
computing  the  eight  ordered  outcome  probabilities  for  each 
event  pair. 

3 . Analysis 

The  next  phase  in  the  demonstration  was  to  replicate 
the  analyses  performed  by  Mitchell  and  Tydeman  as  e-xtended 
for  this  application.  This  required  that  the  responses 
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obtained  be  converted  into  conditional  probabilities,  and 
that  these  be  used  to  construct  a cross-event  matrix  and 
generate  scenarios.  This  section  will  describe  the  compu- 
tational procedure  used  and  discuss  the  results  obtained, 
a.  Computational  Procedure 

Once  the  respondents'  estimates  were  averaged 
and  the  standard  deviations  for  each  relationship  computed, 
mean  values  were  used  to  compute  the  ordered  outcome  proba- 
bilities. The  through  Pg  values  for  each  event  pair 
were  calculated  using  the  formula  derived  by  Mitchell  and 
Tydeman  as  presented  in  Section  E of  Chapter  Three.  These 
values  were  used  to  compute  the  conditional  probabilities, 
however,  unlike  the  Mitchell  and  Tydeman  demonstration, 
every  event  pair  was  found  to  have  inconsistent  results. 

According  to  Mitchell  and  Tydemand  inconsistency 
exists  when  any  outcome  probability  is  negative  [Ref.  29, 
p.  142] . In  fact  pairwise  inconsistency  from  a probabilistic 
standpoint  is  manifested  by  a value  >1  or  ^0,  and  is  indi- 
cated when  the  common  factors  a,  i,  and  v exceed  unity  or 
become  negative.  The  calculated  outcome  probabilities  for 
each  event  pair  are  presented  in  Appendix  B. 

The  cross-event  procedure  uses  a minimum  stan- 
dard deviation  fitting  technique  to  resolve  pairwise  incon- 
sistency. This  technique,  as  described  in  Chapter  Three, 
was  felt  to  be  inappropriate  for  use  in  this  demonstration. 
One  reason  was  that  it  would  require  an  excessive  amount  of 
time  to  resolve  fifteen  inconsistent  sets.  Also,  the 


technique,  as  demonstrated  by  the  developers  in  a case 
resolving  individual  inconsistency,  zeroed  the  inconsistent 
values.  Of  the  fifteen  event  pairs  developed  in  this  study, 
over  half  the  outcomes  in  twelve  pairs  were  inconsistent 
while  six  pairs  had  seven  or  more  inconsistent  values.  As 
a result,  it  was  felt  that  the  fitting  technique  would 
destroy  too  much  potentially  useful  information. 

Even  though  the  ordered  outcome  probabilities 
were  apparently  inconsistent,  it  was  still  possible  to  com- 
pute conditional  probabilities  using  the  formulae  devised 
by  Mitchell  and  Tydeman.  These  conditional  probabilities 
would  doubtlessly  also  be  inconsistent  from  a probabilistic 
standpoint,  however,  in  the  cases  where  one  conditionaj. 
probability  appeared  consistent  it  would  be  possible  to 
employ  Enzer's  boundary  rule  to  resolve  the  second  proba- 
bility. As  in  the  case  of  the  fitting  technique,  this  pro- 
cedure was  expected  to  destroy  at  least  half  of  Lhe  potential 
information.  In  addition,  the  boundary  rule  could  not  be 
applied  in  cases  where  both  conditional  probabilities  appeared 
to  be  inconsistent. 


Because  the  demonstration  results  were  apparently 
inconsistent,  it  was  not  possible  to  produce  all  the  data 
required  for  the  intended  analysis  and  evaluation.  Having 
only  the  ordered  outcome  probabilities  precluded  any  attempt 
to  determine  event  dominance  and  sensitivity.  Nor  was  the 
required  data  produced  for  completing  the  cross-event  matrix 
and  undertaking  scenario  generation.  Therefore,  it  was  not 
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possible  to  conduct  the  analyses  which  use  these  tools.  In 
particular,  the  demonstration  could  not  be  used  to  determine 
the  validity  of  the  proposed  extension  which  included  dis- 
crete policy  options  in  the  event  matrix.  Even  so,  it  is 
still  felt  that  this  was  a practicable  extension,  given  a 
workable  cross-impact  format,  which  could  become  a useful 
analytical  tool. 

Two  alternatives  for  solving  the  inconcsistency 
problem  seemed  to  present  themselves.  The  designed  study 
could  be  continued  by  arbitrarily  specifying  a consistent 
set  of  values.  This  alternative  would  have  ignored  the 
study's  intended  purpose  of  assessing  the  utility  and 
effectiveness  of  cross-impact  analysis  as  a possible 
decision-aid  by  forcing  the  methodology  to  work  rather  than 
examining  its  problems  and  their  implications. 

The  second  alternative  was  to  examine  the  appar- 
ent inconsistencies  to  determine  if  they  were  in  fact  incon- 
sistent. It  would  then  be  possible  to  determine;  a)  why 
the  information  was  inconsistent  and  what  corrective  measures 
could  be  taken,  or  b)  why  the  information  appeared  to  be 
inconsistent,  and  how  it  could  be  used  if  it  was  in  fact 
consistent.  This  alternative  was  selected.  The  approach 
used  will  be  described  in  the  following  section. 

b.  Exploring  the  Inconsistency  Dilemma 

The  first  task  in  e.xploring  the  inconsistency 
problem  was  to  determine  if  the  demonstration  results  were 
in  fact  inconsistent.  To  do  this  the  cross-event  approach's 
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assumptions  and  implications  were  re-examined.  It  was 

obvious  that  the  collected  data  provided  enough  information 

to  rank  the  ordered  outcome  probabilities.  If  the  outcome 

probabilities  were  considered  as  an  eight  element  set  there 
8 N ' 

were  Pg  = Possible  rank  combinations.  If  it 

could  be  shown  that  the  fifteen  combinations  obtained  during 
the  demonstration  formed  a pattern  displaying  some  underlying 
logical  consistency  apart  from  the  consistency  demanded  by 
the  constraints  of  probability  theory,  and  if  this  pattern 
occurred  in  a statistically  significant  number  of  cases, 
then  the  responses  could  be  demonstrated  to  provide  useful 
information  which  might  otherwise  be  masked  by  routine 
cross-impact  analysis  techniques. 

The  respondents  had  been  asked  for  causal, 
time-dependent  estimates  which  were  used  to  compute  ordered 
outcome  probabilities.  The  estimates  were  obtained  by 
assuming  the  certain  occurrence  or  non-occurrence  of  an 
event  (cause)  and  assessing  the  change  in  probability  this 
produced  in  another  event's  probability  (effect).  The  out- 
come probabilities  computed  using  these  estimates  were  time 
sequences,  which  retained  this  cause  and  effect  structure. 

The  eight  ordered  outcomes  could,  therefore,  be  paired  by 
opposite  cause. This  resulted  in  pairing  the  following 


Actually,  several  pairing  methods  were  possible.  For 
exeunple:  Transpose  (i,j  with  j,i)j,  opposite  cause  (i,j  with 

i»j)/  opposite  effect  (i,j  with  i,j),  and  opposite  cause  and 
effect  (i,j  with  i,j).  However,  since  the  relationship  being 
estimated  assumed  known  cause  (prior  occurrence  or  non- 
occurrence) , in  effect  holding  cause  constant,  the  opposite 
cause  pairing  was  considered  more  relevant  for  testing. 
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outcomes:  (i,j  with  I,  j ) , P^  - (i,3  with  I,J). 

Pj  - P^  (j,i  with  J, i),  and  P^  - Pg  (j,!  with  J,T) . If 
the  respondents  had  been  consistent,  it  could  be  argued 
that  the  pairing  of  probability  ranks  by  cause  would  reveal 
some  logical  pattern.  To  accept  this  argument,  the  pattern 
would  have  to  appear  more  frequently  than  could  be  expected 
by  random  chance. 

Upon  pairing  the  probability  rankings  in  the 
fifteen  cases  (one  for  each  event  pair)  by  cause,  several 
patterns  were  noted.  These  are  depicted  in  Appendix  B. 

As  stated  above  8!  combinations  are  possible.  To  show  con- 
sistency a pattern  would  have  to  permit  prediction  (i.e., 
by  knowning  one  outcome  probability's  rank  it  would  be 
possible  to  determine  its  pair's  rank.)  By  this  criterion 
one  obviously  consistent  pattern  was  noted  and  termed 
the  "Chinese  Box". 

The  "Chinese  Box"  pattern  paired  the  first  and 
last,  second  and  seventh,  third  and  sixth  and,  fourth  and 
fifth  ranked  probabilities.  The  pattern  was  intuitively 
pleasing  as  it  indicated  that  the  respondents  had  ranked 
last  the  opposite  cause  to  the  cause  ranked  first,  ranked 
next  to  last  the  opposite  cause  to  the  cause  ranked  second, 
and  so  forth.  This  pattern  implied  that  some  logical  con- 
sistency e.xisted  in  the  way  respondents  linked  cause  and 
effects  in  the  demonstration  situation.  A second  consistent 
pattern  could  also  be  identified  although  it  was  intuitively 
less  rigorous.  This  was  actually  a family  of  patterns  which 


existed  when  one  pair  and  its  opposite  pair  were  located 
in  the  upper  half  and  lower  half  of  the  scale  respectively. 
This  also  implies  some  degree  of  consistency  although  it 
does  not  allow  exact  prediction.  The  "Chinese  Box"  pattern 
was  selected  for  testing  because  it  was  more  rigorous,  and 
could  be  expected  to  occur  less  frequently  by  random  chance 
than  the  simple  upper-lower  split. 

A "Chinese  Box"  occurs  when  one  of  the  outcome 

pairs  is  ranked  in  position  one  through  four  before  its 

pair  is  ranked  in  the  corresponding  position  from  the  bottom 

4 4 ' 

up  (eight  to  five).  Thus  there  are  ^4  ~ ^ combinations 

which  produce  this  pattern.  But  since  can  be  replaced 
by  P2  b®  replaced  by  P^,  P^  can  be  replaced  by  P^, 

and  Pg  can  be  replaced  by  Pg,  the  actual  number  of  combina- 
tions is  given  by  4!  x2x2x2x2=  384.  Thus  the 
theoretical  probability  of  observing  a "Chinese  Box"  can 
be  determined  as  384/8!  = .0095  = .01.  During  the  demon- 
stration the  following  results  were  obtained.  In  the  fif- 
teen cases,  eight  obviously  inconsistent  sets  occurred  and 
seven  apparently  consistent  sets  were  observed.  This  pro- 
duces an  experimental  frequency  of  .47.  Two  tests  were 
used  to  determine  if  this  result  was  statistically  signifi- 
cant. 

The  Kolmogorov-Smirnov  one  sample  test  provided 
the  following  results  for  a sample  set  of  N = 15  cases, 
and  two  categories  (consistent,  inconsistent)  [Siegel,  1956, 
p.  47-52]. 
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r 


Fo(X) 


Sj^5(X) 


the  theoretical  cumulative  distribution 
of  cases 

the  observed  cumulative  frequency  dis- 
tribution of  a random  sample  of  15  cases 

the  maximum  deviation  between  F (X)  and 
Sj_5(X)  or  IF^(X)  - S^5(X)  | ° 


The  critical  value  for  D at  the  .01  significance  level  is 
.404  for  15  cases  [Siegel,  1956,  Table  E,  p.  251]. 


TABLE  FIVE 


CONSISTENCY  PATTERNS  IN  15  CASES 


OUTCOME  EVALUATED  AS 


f = number  of 
cases  in 
each  category 


Fo(X) 


CONSISTENT 


(X) 


D = |Fq{X)  - Sj_5{X) 


6.85/15  = .456 


Since  D = .456  > .404,  this  finding  is  evaluated  as  signifi- 
cant at  the  .01  level.  However,  as  in  most  cases  involving 
a small  N this  conclusion  would  not  be  justified  if  the 
number  of  consistent  outcomes  was  reduced  by  one.  For  si.x 
consistent  outcomes  D = .39  and  is  only  significant  at  the 
.05  level  where  the  critical  D is  .338. 

As  an  additional  check  on  the  statistical  signi- 


ficance of  this  outcome  and  in  order  to  determine  the  probability 


that  this  demonstration  represents  a random  sample,  the 
Binomial  Test  was  applied  [Siegel,  1956,  p.  36-42].  In 
this  test: 

P » proportion  of  cases  expected  to  be  consistent 

Q » 1 - P = proportion  of  cases  expected  to  be 

inconsistent 

N = number  of  cases 

X = number  of  observed  consistent  outcomes 

The  probability  of  observing  exactly  seven  consistent  out- 
comes is  given  by: 


P (x) 


'x 


P(7)  = (.01)'^(.99)® 


where 


.0000000001 


15! 

TiTT 


Thus  p(7)  = 0,  and  the  probability  of  observing  five  or  more 
consistent  outcomes  can  be  computed  similarly  as  p(x  ^ 5) 

* .0000004.  Since  these  tests  both  indicate  that  the  observed 
frequency  of  consistent  rankings  (.47)  was  significant, 
the  following  conclusion  was  reached. 

Although  all  event  pairs  were  evaluated  as  being 
inconsistent  using  the  cross-event  approach's  probabilistic 
criteria  (any  through  Pg  < 0 or  > 1)  , in  actuality  seven 
cases  were  consistent  when  the  probabilities  were  ranked 
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and  paired  by  opposite  cause.  The  implications  of  this 
finding  will  be  addressed  in  the  concluding  chapter. 

D.  CONCLUSIONS 

This  chapter  described  the  implementation  process  de- 
signed to  demonstrate  cross-event  analysis  in  a hypothetical 
crisis  situation.  Several  problems  were  encountered  in 
putting  the  design  into  effect.  During  the  estimation 
phase  it  was  discovered  that  a "brainstorming"  session 
could  get  out  of  control  and  range  far  from  the  intended 
situation,  or  intimidate  some  respondents.  It  also  seemed 
that  the  respondents  had  difficulty  thinking  in  terms  of 
discrete  events.  Furthermore,  the  method  suggested  for  con- 
ducting "event  assessment"  was  found  to  be  inappropriate 
because  it  used  two  different  scales  and  a multiplicative 
index. 

Other  problems  were  encountered  in  estimating  the 
interrelationships  between  event  pairs.  Some  problems  arose 
due  to  limitations  in  the  implementation  design,  specifically, 
the  problems  of  monotony  and  imprecise  question  wording. 
Because  inconsistent  probabilities  were  obtained,  addi- 
tional problems  were  suspected  to  exist  in  the  estimation 
process.  These  problems  involved  the  sequence  in  which  the 
estimates  were  asked  for  and  the  fact  that  respondents  might 
have  lost  track  of  their  previous  responses  in  the  course 
of  completing  the  forms.  Finally,  the  way  in  which  rela- 
tionships were  quantified  was  suspect. 
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To  overcome  these  problems,  several  solutions  were 
proposed.  An  experienced  facilitator  was  recommended  to 
control  the  brainstorming  process.  In  addition,  incorporating 
a method  for  respondent  self-evaluation  was  suggested  to 
permit  weighting  estimates  for  reliability.  Because  respon- 
dents apparently  had  difficulty  conceptualizing  discrete 
events  it  was  suggested  that  they  thought  in  terms  of  inter- 
relations or  contingencies. 

Improvements  to  the  "event  assessment”  process  included, 
using  similar  scales  for  the  measures  being  combined,  and 
adopting  an  additive  impact  index  rather  than  a multiplici- 
tive  one.  It  was  also  noted  that  further  research  into  the 
components  which  constitute  an  event's  impact,  and  their 
relative  importance  was  also  desirable  in  order  to  devise 
an  appropriate  algorithm  for  combining  the  various  measures. 
Overcoming  the  problems  of  monotony  and  imprecise  question 
wording  in  assessing  interactions  requires  devising  an 
improved  question  format. 

This  improvement  could  be  realized  utilizing  an  inter- 
active computer  program.  This  would  also  allow  respondents 
to  review  their  estimates,  and  could  be  extended  to  provide 
probability  bounds  for  estimates,  and  to  present  respondents 
with  a display  of  the  results  obtained  from  their  inputs. 

Such  programs  have  been  demonstrated  in  several  applications 
reviewed  in  this  study.  The  other  problems  suspected  in  the 
estimation  process  were  only  hypothesized  and  could  not  be 
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verified  without  additional  study.  Data  regarding  question 
sequence  and  respondents'  thought  processes  could  be  col- 
lected while  using  the  interactive  computer  program. 

Once  the  implementation  problems  were  discussed  and 
possible  solutions  suggested,  the  experiment  results  were 
presented  and  analyzed.  It  was  found  that  the  ordered  out- 
come probabilities  for  each  event  pair,  as  computed  from 
the  respondents'  estimates  using  the  rules  of  probability 
theory,  contained  inconsistent  values.  However,  when  the 
information  required  from  the  respondents  was  presented  as 
causal,  time-dependent  estimates,  and  the  information  con- 
tained by  the  computed  output  was  presented  as  the  respon- 
dents' view  of  the  sequence  in  which  the  two  paired 
events  were  likely  to  occur,  another  interpretation  was 
possible. 

The  ordered  outcome  values  could  be  ranked  and  paired 
according  to  opposite  cause.  In  seven  of  the  fifteen 
cases,  this  procedure  produced  a pattern  suitable  for  exact 
prediction.  This  "Chinese  Bo.x"  patter’;  v >3  shown  to  be 
intuitively  pleasing  as  an  indication  that  the  respondents 
had  been  logically  consistent  in  their  responses. 

In  order  to  determine  if  this  result  was  statistically 
significamt,  two  tests  were  applied  to  the  data.  The 
Binomial  Test  showed  that  the  probability  of  obtaining  more 
than  five  "Chinexe  Boxes"  was  close  to  zero,  while  the 
Kolmogorov-Smirnov  one-sample  test  showed  that  the  results 
obtained  were  significant  at  the  .01  level.  Therefore,  it 
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V.  CONCLUSION 


A.  INTRODUCTION 

Aside  from  the  several  minor  problems  encountered  in 
implementing  the  selected  cross-impact  analysis  technique 
in  a hypothetical  situation,  this  study  revealed  a major 
conceptual  limitation  in  the  technique.  This  conceptual 
problem  underlies  a basic  assumption  which  must  be  adopted 
in  any  cross- impact  approach.  The  problem  stems  from 
determining  what  interpretation  should  be  applied  to  the 
estimates  provided  by  the  expert  respondents.  This  problem 
relates  to  determining  how  individual  thought  processes 
actually  work,  and  how  complex  relationships  are  concep- 
tualized. The  problem  underlies  any  methodology  which 
attempts  to  employ  subjective  estimates. 

In  reviewing  cross-impact  analysis ' basic  concepts 
and  assumptions  it  was  noted  that  several  interpretations 
could  be  applied  to  subjective  estimates,  depending  on  the 
assumption  made  concerning  the  way  individual ' s interpret 
and  analyze  unfolding  situations.  These  alternative  inter- 
pretations can  be  grouped  into  at  least  two  main  categories. 
The  first,  which  was  pursued  in  this  study,  viewed  the 
estimates  as  causal  probabilities.  Under  this  interpreta- 
tion, individuals  are  assumed  to  be  essentially  Bayesian 
information  processors. 


The  second  interpretation  views  subjective  estimates 
as  quantified  variation  estimates,  or,  as  directly  describing 
how  a change  in  one  variable  produces  a change  in  another 
variable.  Under  this  interpretation,  an  individual  is 
assumed  to  be  a more  complex  information  processor  than  is 
described  by  the  Bayesian  model.  This  interpretation  was 
used  in  the  Turoff  approach,  which  represented  the  results 
obtained  from  the  respondents'  estimates  as  analogous  to 
correlation  coefficients. 

How  the  question  of  interpreting  subjective  estimates 
is  resolved  forms  the  basis  for  the  methodological  approach 
used  in  cross-impact  analysis.  The  assumed  interpretation 
determines  how  inconsistency  is  identified  and  corrected. 

It  also  influences  the  method  used  to  obtain  estimates  and 
produce  cross-impact  factors.  For  these  reasons,  determining 
the  appropriate  assumption  to  be  used  in  interpreting 
estimates  is  essential  to  performing  reliable,  useful 
cross-impact  analysis. 

The  results  obtained  in  this  study  indicate  that  a 
probabilistic  interpretation  of  estimates  may  not  be  appro- 
priate. It  was  demonstrated  that,  although  the  ordered 
outcome  values  were  inconsistent  as  probabilities,  a 
significant  number  of  event  pairs  contained  an  outcome 
pattern  which  indicated  that  a consistent  thought  process 
had  occurred.  Additionally,  during  event  generation, 
respondents  showed  a tendency  to  think  in  terms  of  contin- 
gencies or  interrelations. 
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This  chapter  will  address  the  question  of  how  respondents 
answer  the  cross-impact  questions  posed  in  light  of  the 
study  results.  This  discussion  will  focus  on  the  alter- 
native interpretations  proposed,  and  how  each  of  them  may 
be  used  in  future  cross-impact  analysis  applications.  The 
final  section  of  this  chapter  will  discuss  the  technique's 

use  as  a decision-aiding  methodology  as  it  is  currently  : 

structured.  This  discussion  will  highlight  the  technique's 

• *1 

strengths  and  weaknesses  in  problem  analysis.  *■  < 

B.  INTERPRETING  RESPONDENT  ESTIMATES 

Of  the  two  interpretations  proposed,  the  most  frequently 
employed  is  probability.  The  inherent  advantages  of  this 
interpretation  were  discussed  in  the  technique  selection 
section  of  Chapter  Three.  These  advantages  were  primarily 
methodological  in  nature.  The  probabilistic  assumption 
provided  a method  for  determining  if  respondents'  estimates 
were  consistent.  Furthermore,  by  using  probabilistic  rela- 
tionships, a well  structured  algorithm  for  manipulating 
estimates  was  also  developed. 

1 . Probabilities 

Using  probability  values  restricted  the  estimated 
and  computed  values  to  a range  between  one  and  zero. 

Several  procedures  were  designed  to  correct  values  which 
exceeded  these  bounds.  All  of  these  procedures  essentially 
zeroed  or  made  unity  the  extreme  values,  .^s  demonstrated 
in  this  study,  use  of  these  correction  procedures  could 
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destroy  much  information.  This  is  true  especially  when  a 
large  number  of  extreme  responses  are  obtained.  This  study 
also  demonstrated  that  bounding  the  estimated  values  to 
normal  probability  limits  can  disguise  otherwise  consistent 
results.  A possible  explanation  behind  the  excessive 
number  of  extreme  responses  is  that  the  respondents  do  not 
easily  conceptualize  events  and  their  interactions  in 
probabilistic  terms. 

This  problem  could  be  reduced  if  respondents  were 
trained  to  make  probability  estimates.  An  interactive 
computer  package  could  aid  in  this  process.  As  discussed 
previously,  such  a package  could  be  structured  to  provide 
respondents  with  the  probability  bounds  for  their  future 
estimates  based  on  the  values  previously  specified.  The 
program  would  also  furnish  the  respondents  with  results 
computed  from  their  estimates.  Repeated  exposure  to  such 
outputs  could  increase  the  respondents'  experience  with 
probabilistic  relationships.  Eventually,  they  might  become 
proficient  in  making  consistent  probabilistic  estimates. 

However,  aside  from  the  previously  mentioned 
methodological  advantages,  there  does  not  appear  to  be  any 
innate  justification  for  pursuing  this  solution.  The  results 
of  this  study  indicate  that  consistent  responses  can  be 
obtained  when  using  other  than  consistent  probability  values. 
If  respondents  do  find  probabilities  difficult  to  work  with, 
yet  can  produce  consistent  responses  through  some  other 
thought  process,  it  might  be  more  appropriate  to  devise  an 
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analytic  procedure  which  taps  expert  opinions  by  using 
some  other  algorithm.  In  this  case,  the  second  inter- 
pretation given  respondents'  estimates  might  be  worthy  of 
increased  attention  and  analysis. 

2 . Variations 

The  second  interpretation  given  respondents' 
estimates  suggests  that  individuals  may  be  more  comfortable 
working  with  a procedure  which  quantifies  the  variations 
produced  by  one  variable  interacting  with  another  variable. 
The  fact  that  the  respondents  appeared  to  think  in  terms 
of  contingencies  during  event  generation  lends  some  support 
to  this  interpretation.  If  it  can  be  determined  that 
respondents  are  more  comfortable  when  dealing  with  relative 
changes  in  events,  a variation  algorithm  might  be  used  to 
collect  and  manipulate  their  estimates. 

Two  measures  have  been  developed  which  describe  the 
relationships  between  variables  (in  this  case  events). 

Either,  or  both  might  be  developed  for  use  in  cross-impact 
analysis.  Turoff  demonstrated  the  application  of  one 
variation  measure,  a correlation-based  algorithm,  in  his 
cross-impact  approach.  The  second  variation  measure, 
regression,  might  also  be  appropriate  for  use  in  cross- 
impact analysis. 

Correlation  coefficients  range  between  minus  and 
plus  one.  These  boundaries  could  produce  constraints  similar 
to  those  encountered  in  using  probabilities.  However,  using 
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a correlation  algorithm  would  retain  a certain  ease  of 
manipulation.  In  addition,  it  would  also  be  possible  to 
obtain  an  indication  of  estimator  consistency. 

This  capability  derives  from  the  fact  that  under 
this  interpretation  the  cross-impact  matrix  would  be 
analogous  to  a correlation  matrix.  Consistency  could 
therefore  be  determined  through  analyzing  the  matrix  entries. 
A brief  example,  using  a simplified  correlation  matrix,  will 
demonstrate  this  point 


FIGURE  9 

SIMPLIFIED  CORRELATION  MATRIX 
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In  a correlation  matrix,  if  vari.ible  C increases 
with  an  increase  in  variable  A;  then  variable  A must  increase 
with  an  increase  in  variable  C.  In  other  words,  the  matrix 
entries  above  the  diagonal  must  mirror  the  entries  below 
the  diagonal  to  be  consistent. 
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As  a final  consideration,  the  theory  describing 
correlation  coefficients  is  well  enough  developed  that 
other  analysis  forms  might  be  attempted.  For  example,  the 
technique  of  path  analysis  could  be  developed  as  a method 
for  tracing  causal  relationships.  It  might  also  be 
possible  to  trace  logical  structures  on  adjacency  matrices 
using  the  principles  developed  by  Harary  [Ref.  42],  and 
demonstrated  by  Bonham,  et  al.,  and  Axelrod  [Refs.  44,  45, 
and  43).  The  correlation  algorithm  for  cross-impacts  might 
also  be  extended  into  a regression  algorithm. 

Regression  coefficients  represent  the  second  varia- 
tion measure  which  might  be  applied  to  cross-impact  analysis. 
Because  regression  coefficients  can  range  between  plus  and 
minus  infinity,  there  would  be  no  constraints  imposed  by 
bounding  estimate  values.  However,  using  a regression 
algorithm  could  easily  produce  extremely  large  values, 
limiting  the  ease  with  which  estimates  could  be  manipulated. 
This  fact  is  apparent  upon  recalling  that  some  ordered  out- 
come values  computed  in  this  study  exceeded  a value  of 
thirty,  and  these  were  calculated  using  bounded  estimates. 

3 . Summary 

This  study's  results  indicate  that  the  method  used 
in  cross-event  analysis  to  tap  expert’s  opinions  may  be 
inappropriate.  Two  alternatives  might  be  used  to  overcome 
this  problem.  The  first  possibility  is  to  train  respondents 
to  think  in  probabilistic  terms.  The  second  is  to  develop 


another  algorithm  for  interpreting  and  manipulating  their 
estimates,  possibly  through  using  correlation  or  regression 
coefficients. 

Choosing  between  these  two  alternatives  cannot  be 
attempted  without  further  study.  The  primary  focus  for 
this  additional  study  would  be  to  empirically  determine  how 
people  actually  conceptualize  relationships  between  events. 
Without  an  understanding  of  this  crucial  point,  selection 
of  a cross-impact  algorithm  can  only  be  made  arbitrarily. 

As  a result,  arbitrary  estimate  interpretation  would 
preclude  giving  any  confidence  to  specific  results  obtained 
from  the  analysis.  Despite  this  problem,  it  is  felt  that 
the  methodology  has  some  potential  as  an  alternative 
approach  to  decision  analysis  in  socio-political 
applications . 

C.  CROSS-IMPACT  ANALYSIS  AS  A DECISION-AID 

Several  advantages  can  be  identified  for  the  cross-impact 
technique  when  employed  in  assessing  complex  situations. 
First,  its  systematic  approach  assures  that  all  problem 
elements  identified  as  important  are  considered.  The  step 
by  step  procedure  used  can  be  easily  traced  to  identify 
important  elements  in  the  problem  situation,  and  to  suggest 
areas  requiring  further  study  or  close  monitoring. 

Secondly,  the  technique  can  serve  as  a valuable  heuristic 
device.  Its  systematic  approach  forces  respondents  to  care- 
fully evaluate  each  estimate  (opinion)  in  conjunction  with 
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their  other  estimates,  and  to  consider  the  estimates' 
implications  in  to to.  This  requires  that  the  experts  fully 
and  consistently  formulate  an  underlying  paridigm  which  can 
aid  an  individual  or  organization  in  problem  definition. 

Thus,  cross-impact  analysis  can  be  a useful  modeling  tool 
and  problem  specification  approach. 

In  addition,  because  the  technique  is  time  sensitive, 
its  results  can  be  used  to  monitor  problem  development,  or 
at  least  the  way  in  which  the  experts  perceive  that  develop- 
ment. Through  monitoring  the  development  of  cross-impact 
matrices  it  might  also  be  possible  to  learn  more  about  how 
individual,  group,  and  organizational  thought  processes 
evolve  in  response  to  evolving  situations,  changing  informa- 
tion streams,  and  differing  perceptions  of  problem  compo- 
nents. Thus,  the  technique's  continued  use  might  contribute 
to  an  understanding  of  how  individuals  process  information 
(providing  the  data  with  which  to  resolve  the  question  of 
how  respondent's  estimates  should  be  interpreted),  and  how 
they  perceive  developing  systems. 

Even  without  having  completed  the  designed  implementa- 
tion, because  inconsistent  ordered  outcome  probabilities 
were  obtained,  it  is  still  possible  to  make  some  observa- 
tions regarding  cross-impact  analysis'  utility  as  a decision- 
aid.  Obviously,  there  are  several  problems  which  must  be 
corrected  before  the  technique  can  be  applied  in  a "real- 
world"  analysis.  Many  problems  encountered  in  the 
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demonstration  arose  from  a lack  of  experience  in  conducting 
subjective  estimation  sessions.  Such  "operator"  problems 
included  controlling  the  brainstorming  session,  and 
conducting  the  "event  analysis.”  These  problems  were 
discussed,  and  solutions  suggested  in  Chapter  Four.  None 
of  the  implementation  problems  appear  to  inhibit  using  the 
technique  as  much  as  the  conceptual  problem  previously 
discussed. 

The  theory  that  interactions  among  discrete  events  can 
be  quantified  and  systematically  assessed  seems  to  be  an 
appropriate  method  for  approaching  complex,  ill-defined 
problems.  The  technique's  demonstrated  computer  adapta- 
bility enhances  this  attractiveness.  Using  a computer 
makes  it  possible  to  identify  a large  number  of  events  for 
analysis.  In  some  cases,  it  might  be  possible  to  practically 
specify  the  entire  problem  environment.  Although  this  would 
require  a very  large  number  of  estimates,  they  could  be 
obtained  and  manipulated  with  the  computer.  However,  until 
the  conceptual  question  discussed  in  the  previous  section 
is  resolved,  cross- impact  analysis  remains  a developmental 
methodology . 
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APPENDIX  A 


ESTIMATED  RESPONSES 


1.  Candidate  event  set,  probability  and  significance 
ratings,  impact  scores  (standard  deviations) 


No. 

Event 

Statement 

Prob 

(S.D.) 

Sig. 

(S.D.) 

Imp. 

(S.D.) 

1 

Kurds  in  Soviet  Union 
revolt. 

.115 

.08 

.48 

.31 

.62 

.27 

2 

Egypt  closes  Suez 

Canal  to  Soviets . 

.315 

.12 

.595 

.22 

.91 

.17 

3 

United  States 
deploys  Carrier  Task 
Force  to  Indian 

Ocean. 

.755 

.19 

.62 

.19 

1.375 

.31 

4 

Soviet  Union 
initiates  military 
alert  along 

Chinese  border . 

.53 

.28 

.71 

.25 

1.24 

.47 

5 

Soviet  Union  block- 
ades Gulf  of  Oman. 

.275 

.19 

.725 

.25 

1.0 

.32 

6. 

Soviet  Union  mines 

St.  of  Hormuz. 

.185 

.14 

.77 

.25 

.955 

.26 

7 

United  States  appeals 
to  all  parties  to 
cease  hostilities. 

.805 

.13 

.18 

.17 

1.04 

.17 

8 

United  States  offers 
to  negotiate  cease 
fire. 

.64 

.18 

.48 

.23 

1.12 

.27 

9 

United  States  evac- 
uates civilians 
from  area. 

.835 

.10 

.305 

.14 

1.135 

.14 

10 

China  ships  arms  to 
Baluchi  rebels. 

.575 

.27 

.575 

.16 

1.15 

.39 

11 

India  lends  moral 
support  to  anti- 
Soviet  forces. 

.685 

.14 

.405 

.22 

1.09 

.29 

No. 

Event 

Statement 

Prob  (S.D.) 

Sig. 

(S.D. 

Iiqp. 

(S.D. 

12 

Chinese  mobilize  in 
Sinkiang. 

.505 

.25 

.715 

.20 

1.22 

.37 

13 

Iranian  leftists 
sabotage  oil  fields. 

.485 

.16 

.715 

.16 

1.20 

.27 

14 

United  States  states 
it  has  no  intention 
of  becoming  involved 
in  the  situation. 

.275 

.10 

.675 

.31 

.945 

.28 

15 

Leftists  seize 
control  in  Iran. 

.37 

.14 

.805 

.14 

1.175 

.13 

16 

United  Nations 
meets  to  discuss 
the  problem. 

.885 

.08 

.195 

.14 

1.08 

.17 

17 

Iranian  and  Soviet 
units  skirmish  in 
area  around 

Afghanistan . 

.485 

.25 

.625 

.20 

1.12 

.37 

18 

Iranian  government 
clamps  down  on 
leftists . 

.71 

.18 

.375 

.13 

1.09 

.13 

19 

Civil  War  breaks 
out  in  Iran. 

.535 

.24 

.595 

.24 

1.22 

.18 

20 

United  States 
convoys  oil  ship- 
ments with  Naval 
units . 

.385 

.25 

.570 

.22 

.955 

.42 

21 

Ayatolla  Khomeini 
calls  for  Jihad. 

.48 

.25 

.545 

.27 

1.03 

.27 

22 

Soviet  backed  Afghani 
forces  subjugate 
Baluchistan. 

.595 

.15 

.885 

.14 

1.48 

.19 

23 

Soviet  Union  air- 
lifts supplies  to 
Afghanistan. 

.72 

.20 

.65 

.17 

1.37 

.34 

24 

United  States  sends 
aid  to  Pakistan. 

.465 

.16 

.60 

.12 

1.07 

.24 

25 

United  States  sends 
aid  to  Iran. 

.425 

.18 

.58 

.21 

1.01 

,37 
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Event 

Statement 


Prob  (S.D.)I  Si' 


(S.D.) 


. (S.D.) 


United  States 
deploys  troops  to 

Saudi  Arabia. 

.255 

.13 

.625 

.17 

.88 

.22 

United  States 
deploys  an  F-15 
squadron  to 

Saudi  Arabia. 

.475 

.23 

.425 

.2 

.90 

.40 

Soviet  Union  builds 
up  its  Indian  Ocean 
squadron . 

.795 

.19 

.625 

.1? 

1.42 

.28 

Soviet  Union 
harrasses  oil  tankers. 

.25 

.20 

.685 

.25 

.935 

.30 

(*  indicates  events  chosen  for  analysis) 
Estimated  interactions  by  Event  Pair 


Event  Pair 
(i/ j) 

P(i)  P(i) 


(i|j) 


'-2 

(j|i) 


(i|j) 


'-4 

(j|i) 


(if li&j) 


(22,28) 

.6  .8 

.66 

.83 

.56 

.80 

.23 

1 (22,23) 

.6  .72 

.72 

.80 

.54 

.70 

1 

.24 

^1 

(ilj) 

^2 

(jli) 

‘=3 

(i|  j) 

^4 

(j|i) 

(if li&j) 

.57 

.85 

.67 

.76 

.51 

.68 

.92 

.59 

.83 

.51 

.58 

.61 

.65 

.68 

.32 

.82 

.77 

.79 

.72 

.56 

.86 

.87 

.80 

.75 

.61 

.81 

.86 

.80 

.84 

.77 

.8 

.66 

.81 

.66 

.74 

.75 

.81 

.73 

.75 

.56 

.73 

.86 

.72 

.83 

.78 

.73 

■ " 

.70 

.73 

.64 

.69 

.79 

.86 

.78 

.83 

.80 

.77 

.65 

.77 

.68 

.52 

82 


66 


85 


67 


35 


2.  Probability  rankings  by  Event  Pair 


(22,28)  Or:  Sov/Afg.  forces  gain  control  of  Baluchistan, 
Soviet  Union  builds  up  I.O.  squadron 

Outcome  Ranking  Meaning  Pairing  by  0pp.  Cause 

1.  P, 

I 

I 3.  P5 

; 4.  Pg 


( 


(22,3)  Or:  Sov/Afg.  forces  gain  control  of  Baluchistan, 
U.S.  deploys  CTF  to  I.O. 


Outcome  Ranking  Meaning 


Pairing  by  0pp.  Cause 


Dif . 


Evaluation: 


Apparently  Inconsistent,  however,  value  difference 
from  rank  7 to  rank  8 is  only  .2;  therefore, 
ossibly  consistent 


(22,9)  Or:  Sov/Afg.  forces  gain  control  of  Baluchistan, 
U.S.  evacuates  civilians  from  area 


Outcome  Ranking  Meaning 


Pairing  by  0pp.  Cause 


1. 

^7 

( j ,i) 

2. 

^8 

( j ,1) 

3. 

(I»  j) 

4. 

^2 

(i,3 ) 

5. 

^3 

(i»  j) 

6. 

^6 

( j ,1) 

7. 

( j »i) 

, 

8. 

(i»  j) 

i 1 1 M M 1 III! 
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t 


(28,3)  Or:  S.U.  builds  up  I.O.  squadron 
U.S.  deploys  CTF  to  I.O. 

Outcome  Ranking  Meaning  Pairing  by  0pp.  Cause 


(28,9)  Or:  S.U.  builds  up  I.O.  squadron, 

U.S.  evaculates  civilians  from  area 

Outcome  Ranking  Meaning  Pairing  by  0pp.  Cause 


^'.on:  Inconsistent 


(28,8)  Or:  S.U.  builds  up  I.O.  squadron 

U.S.  offers  to  negotiate  a cease-fire 


Outcome  Ranking  Meaning 


1. 

^7 

( j ri) 

2. 

^8 

( j »I) 

3. 

( j ,1) 

4. 

^5 

( j /i) 

5. 

^2 

(i/ D) 

6. 

"l 

(i/ j) 

7. 

(!,]) 

8. 

^3 

(I,j) 

Evaluation : 

Inconsistent 

Pairing  by  0pp.  Cause 

□ 


(23,3)  Or:  S.U.  airlifts  supplies  to  Afghanistan, 

U.S.  deploys  CTF  to  I.O. 

Outcome  Ranking  Meaning  Pairing  by  0pp.  Cause 


1. 

(if  j) 

2. 

"5 

( j f i) 

3. 

^6 

(jfl) 

4. 

"2 

(ifj) 

5. 

(I, : ) 

6. 

^8 

( j f I) 

7. 

(If  j) 

8. 

(If  i) 

.07 


Evaluation:  Apparently  Inconsistent,  however,  value  difference 

from  rank  7 to  rank  8 is  only  .07,  therefore. 
Consistent . 


(23,9)  Or;  S.U.  airlifts  supplies  to  Afghanistan, 
U.S.  evacuates  civilians  from  area 


Outcome  Ranking  Meaning 


1. 

^1 

(i/ j) 

2. 

^5 

( j »i) 

3. 

( j ,i) 

4. 

^2 

(if  3) 

5. 

^6 

(jfi) 

6. 

^8 

(3/1) 

7. 

(I/j) 

8. 

^3 

(1/ j) 

Evaluation; 

Inconsistent 

Pairing  by  0pp.  Cause 

II 

□1 


(23,8)  Or:  S.U.  airlifts  supplies  to  Afghanistan, 
U.S.  offers  to  negotiate  a cease-fire 


Outcome  Ranking  Meaning 


Pairing  by  0pp.  Cause 


1. 

^1 

(if  j) 

2. 

^5 

( j f i) 

3. 

^2 

(ifj) 

4. 

^6 

( j f I) 

5. 

^8 

(jf  I) 

6. 

^4 

(1/3) 

7. 

^3 

(ifj) 

8. 

(3/ i) 

Evaluation; 

Apparently  I 

iDif . I = 


difference  between  rank  7 and  rank  8 is  only 
.01,  therefore,  Consistent. 


.01 
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W 


(3,9)  Or;  U.S.  deploys  CTF  to  I.O. 

U.S.  evacuates  civilians  from  area 


Outcome  Ranking  Meaning 


Pairing  by  0pp.  Cause 


1. 

^1 

(io) 

2. 

^7 

(j  ,i) 

3. 

in 

( j »i) 

4. 

to 

(i/D) 

5. 

00 

04 

( j /I) 

6. 

^6 

( j /I) 

7. 

(1/5) 

8. 

^3 

(1/ j) 

□ 


□ 


(3,8)  Or;  U.S.  deploys  CTF  to  I.O., 

U.S.  offers  to  negotiate  a cease-fire 


Outcome  Ranking  Meaning 


Pairing  by  0pp.  Cause 


j 


9,8  Or:  U.S.  evacuates  civilians  from  area, 

U.S.  offers  to  negotiate  a cease-fire 


Outcome  Ranking  Meaning 
L.  P5  (j,i) 

1.  (i,j) 

5.  P2  (ifD) 

1.  Pg  (j»I) 

j.  Po 


(1,5) 


7.  P3  (I,j) 

8.  Pj  (j,i) 

Evaluation:  Consistent 

3.  Summary  of  Results 


Case 


Pair 


Consistent 


1 

(22,28) 

NO 

2 

(22,23) 

YES 

3 

(22,3) 

NO 

4 

(22,9) 

NO 

5 

(22,8) 

YES 

6 

(28,23) 

YES 

7 

(28,3) 

YES 

8 

(28,9) 

NO 

9 

(28,8) 

NO 

10 

(23,3) 

YES 

11 

(23,9) 

NO 

12 

(23,8) 

YES 

13 

(3,9) 

NO 

14 

(3,8) 

NO 

15 

(9,8) 

YES 

Pairing  by  0pp.  Cause 


Order 


7,  8,  5 
5,  6,  1 
1.  5.  6 


All  inconsistent  outcome  orders  are  unique. 

The  seven  consistent  outcome  patterns  occur  in  5 sets. 
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